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ABSTRACT 


This thesis research examines the effectiveness of a newly developed 
cognitive and pedagogical evaluation framework to assess computer-based 
instruction. All training programs must have comprehensive evaluation guidelines 
in place to ensure the quality of instruction from the classroom-training 
environment to the virtual training environment is not diminished. The application 
of sound cognitive and pedagogical principles helps ensure that an organization’s 
training goals will be met. This research developed a set of practical guidelines, 
or a template, that should be used to evaluate the cognitive and pedagogical 
aspects of any given computer delivered course of instruction. This template is 
used to evaluate the United States Navy’s newly developed CD-ROM Surface 
Warfare Officer (SWO) training course. The SWO course is the basic 
professional training for jjnior Ensigns that is now contained on CD-ROM and 
delivered by personal computer. 
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EXECUTIVE SUMMARY 


This thesis research examines the effectiveness of a newly developed 
cognitive and pedagogical evaluation framework to assess computer-based 
instruction. All training programs must have comprehensive evaluation guidelines 
in place to ensure the quality of instruction from the classroom-training 
environment to the virtual training environment is not diminished. 

Chapter II provides background information on human sensory perception, 
short and long term memory, problem solving, and learning. Chapter III is a 
review of sound instructional and pedagogical principles for developing 
Computer-based Training (CBT) and evaluative framework development for 
assessing CBT initiatives. Chapter IV is the analysis of the strengths and 
weaknesses of the United States Navy’s newly developed CD-ROM Surface 
Warfare Officer (SWO) training course. The SWO course is a basic professional 
course of instruction for junior Ensigns that is mostly contained on CD-ROM and 
delivered through the use of a personal computer. Chapter V is the 
recommendations that arise out of the analysis conducted in Chapter IV. 



THIS PAGE INTENTIONALLY LEFT BLANK 



I. INTRODUCTION 


As the use of computers to enhance and replace traditional classroom 
instruction increases, evaluation guidelines must be developed and implemented 
to ensure that the quality of instruction is not diminished. Much has been written 
about why and how to successfully implement organizational computer-based 
training systems. However, little has been mentioned about how to evaluate 
whether specific cognitive and pedagogical principles are being effectively 
employed. The application of cognitive and pedagogical principles helps to 
ensure that the organization’s training goals will be met. 

This research developed a set of practical guidelines, or a template, that 
should be used to evaluate the cognitive and pedagogical aspects of any 
computer delivered course of instruction. In this thesis, the template is used to 
evaluate the United States Navy’s newly developed CD-ROM Surface Warfare 
Officer (SWO) course. The SWO basic course is the first professional course of 
instruction for junior Ensigns that is mostly contained on CD-ROM and delivered 
through by personal computer. 

A. BACKGROUND 

In 2002 the United States Navy’s Surface Warfare Officer (SWO) 
community dramatically changed the way its newly commissioned junior officers 
receive their introductory specialty training. The SWO community has shifted the 
delivery method from traditional classroom and laboratory training to a Compact 
Disk Read Only Memory (CD-ROM) distance-learning environment that is 
supplemented by on-the-job training (OJT). The Navy awarded the contract to 
re-engineer the then current curriculum to Intelligent Decision Systems, Inc (IDSI) 
in mid-2002. The delivery date was December 2002. In January 2003, all newly 
commissioned Ensigns in Surface Warfare started reporting directly to a ship, 
vice the Naval Education and Training Center, Newport, Rhode Island, and were 
issued the new courseware, and the commencement of warfare specialty 
training. 
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B. PROBLEM STATEMENT 


Human perception, memory, and critical thinking skills are all cognitive 
processes that must be considered by anyone that is involved in developing 
courses of learning, whether those courses are taught by a human instructor or 
learned in a virtual training environment. According to Filbert and Weatherspoon 
(1993), designers of interactive learning systems must take into account the 
appropriate cognitive learning theories and instructional design paradigms prior 
to constructing their systems, so the learning systems that are developed will 
have instructional effectiveness (p.1). 

Just as it is important to properly design instructional systems, it is 
important to design and implement effective evaluation guidelines. Without 
effective evaluation, an organization will not know if its training objectives are 
being adequately met. Furthermore, organizational leadership may not know if 
the training methods that are being employed are the best techniques available 
to maximize student learning throughout the organization’s training domain. 

Due to the short duration between the letting of the contract and the date 
that deliverables were due, IDSI had to hire subcontractors, and utilize off-the- 
shelf products where available and applicable. However, it is possible that by 
doing so, the program emphasis was on building a working course of instruction 
rather than building a course of instruction that would best fit the students needs. 

Consequently, if the courseware (instructional material and delivery 
method) does not adequately meet the students’ needs, then it does not meet the 
Navy’s needs. If the Navy’s needs are not being met by the CD-ROM training 
program, then other alternatives, including the possibility of returning to previous 
methods of instruction need to be identified and acted upon. 

C. SCOPE OF THESIS 

The scope includes: (1) A relevant review of human memory, cognitive 
models, and learning theories, (2) a review of current methods that are used in 
the arena of Information Technology and Distance Learning to implement an 
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effective computer-based training environment, (3) development of an evaluative 
framework that is sufficient to evaluate the cognitive and pedagogical aspects of 
Computer-Based Training, (4) completion of a cursory analysis of the United 
States Navy’s Division Officer classroom-based course of instruction for Surface 
Warfare Officers and an in-depth analysis of the reengineered CD-ROM based 
curriculum and methods of delivery. The thesis will conclude with predictions for 
the eventual success or failure of the redesigned curriculum where the use of 
Information Technology has been implemented and will offer suggestions, if any, 
for the improvement of the pedagogy and curriculum. At this time there are no 
objective data available for comparing computer-based to conventional 
instruction. Accordingly, the course assessment is based primarily on 
instructional quality assessment measures, instructional design and computer- 
based delivery pedagogical guidelines. 

D. LIMITATIONS 

In its Division Officer at Sea Training Program Overview, IDSI identifies 
seven key elements that comprise the SWO DIVO At Sea training program. 
Those elements are an Individual Development Plan (IDP), Interactive 
Courseware (ICW), Practicums, Practical Problems, Case Studies, Shipboard 
Experiences, and Assessment (p. 3-1). IDSI’s overview emphasizes four 
instructional categories, namely ICW, Practicums, Practical Problems, and Case 
Studies, which are important to the successful implementation of this training 
endeavor. However, this thesis will primarily focus on the curriculum integration 
strategy, and in particular, a critical analysis of the instructional strategies and 
delivery methods employed to create the SWO division officer interactive 
courseware. 

E. METHODOLOGY 

In order to evaluate whether a course of instruction meets its objectives, 
one must be aware of the cognitive and pedagogical principles that must be 
satisfied in order to ensure that learning occurs. Consequently, this thesis first 
examines leading cognitive and pedagogical models and theories of learning. A 
generic evaluative model for evaluating computer-based instructional delivery 
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systems is developed and presented. Although developed primarily to analyze 
computer-based instructional systems, with minor modifications, it is useful in 
evaluating traditional educational environments as well. Next, the traditional 
(classroom) Surface Warfare Officer Curriculum is briefly examined and then the 
evaluation template is used to thoroughly evaluate the relative strength and 
weaknesses of the reengineered SWO courseware against existing cognitive and 
pedagogical principles. Finally, the evaluation results are summarized and all 
conclusions and recommendations presented. 

Qualified Surface Warfare Officers serving as training officers aboard their 
respective ships completed surveys developed for supervisors. These officers 
have all successfully passed the Surface Warfare course of instruction at 
Newport, Rhode Island, and having mastered the skills that Ensigns will be 
required to learn, have earned the right to wear the Surface Warfare Pin. 
Ensigns that are currently enrolled in the new SWO basic course completed 
surveys that were devebped for students. The Ensigns were used because it is 
important to discover how individuals who are actually taking the computer-based 
course feel about the way it is being delivered. It cannot be overemphasized that 
if the students’ needs are not being met, then they are probably not learning to 
the best of their potential, and consequently, there is a high probability that the 
organization’s training objectives are not being met. 

F. ORGANIZATION OF PAPER 

Chapter II examines memory, cognition, and a review of the major 
theories of learning. Chapter III details the basics of a typical distance-learning 
program and develops a model for the successful evaluation of a computer- 
based course of instruction. Chapter IV reviews various aspects regarding the 
traditional SWO classroom-based course of instruction and evaluates the 
cognitive and pedagogical aspects of the reengineered SWO CD-ROM based 
courseware. Chapter V presents conclusions and recommendations for 
improvements to this new course of instruction. 
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II. MEMORY, COGNITION, AND LEARNING 


Human perception, memory, and cognition play an important role in CBT 
programs. To effectively develop learning materials for students, course 
designers must take the way people learn into account. To varying degrees, 
mankind processes information similarly. There are, of course, disputes among 
theorists about how this process works, but that does not change the fact that 
human beings sense, process, synthesize, remember, and recall information in 
the same way. Snce psychology has such a large impact on education and 
training, it is appropriate to include information about memory, cognition, and 
human learning in any research that is related to CBT. 

A. MEMORY 

For the purposes of this paper, human memory is the ability to store, 
process, retain, and recall information over the course of a lifetime. Memory is 
tremendously important. Dix, Finlay, Abowd, and Beale (1997) demonstrate that 
most of our everyday activities rely on memory, either in the storage of actual 
facts, or the knowledge of actions or procedures. Our memory allows us to 
repeat actions, use language, and utilize new information that is received via our 
senses (p. 26). Humankind is endowed with the ability to perceive our 
environment. This perception comes as a result of taste, smell, sight, hearing, 
and touch, which are the five senses. 

1. Senses 

Wickens (1992) states each one of the five human sensory systems is 
equipped with a central mechanism that prolongs the representation of a physical 
stimulus for a short period of time after the stimulus has stopped (p. 17). This is 
known as the Short Term Sensory Store (STSS). Additionally, Wickens (1992) 
and Dix et al (1997) contend that when a person’s attention is diverted, the STSS 
acts as a “stimuli” buffer, and permits environmental information to be temporarily 
preserved so that it can be dealt with later. According to Dix et al. (1997), a 
STSS exists for each sensory channel: iconic memory for visual stimuli, echoic 
memory for aural stimuli, and haptic memory for touch p. 17). 
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Wickens (1992) relates three general characteristics of the STSS: 

(1) The STSS is preattentive. This means no conscious attention is required to 
prolong the image during the natural “time constant” of the sensory store (p. 18). 

(2) The STSS is relatively veridical, preserving most of the physical details of the 
stimulus (p. 18). 

(3) The information in the STSS decays rapidly. The iconic store holds sensory 
data for less than one second. The echoic and haptic stores generally hold 
information between two and eight seconds (p. 18). 

2. Short-Term Memory 

Information that is resident within a STSS must be acted upon if it is to be 
remembered. Anderson (2000) shows information that was left “unattended” in a 
STSS was quickly lost. Attended, or acted upon information, went into short-term 
memory (p. 172). Short-term memory has become synonymous with working 
memory. Anderson (2000) and Breuer (1990) also demonstrated if short-term 
memory was not acted upon, or rehearsed, it v\es quickly forgotten as well. Dix 
et al. (1997) believe working memory acts as a “scratchpad” for the temporary 
recall of information (p. 28). 

Short-term memory has a limited capacity. Anderson (2000) reports the 
number of elements in a sequence that a person can ordinarily repeat back 
without error is between seven or eight (p. 17). For example, a person can 
usually remember a seven-digit phone number quite easily. However, if that 
phone number includes an area code, bringing the number of digits to ten, it 
often becomes necessary to write the number on a piece of paper and rehearse 
it before it is recalled correctly. Anderson (2000) believes the number of 
rehearsals control the amount of information that is transferred to long-term 
memory (p. 17). In fact, Anderson (2000) argues that study time and the amount 
of practice with a given data set directly contributes to a person’s ability to 
accurately recall that data when it is needed. Figure 1 shows the relationship 
between senses, short-term memory, and long-term memory. 
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Figure 1. Sensory to Long-term memory (From Dix et al., 1997, p. 27) 

CBT design should take into account the need to act on information 
multiple times so it can be sent from STSS through short-term memory to long¬ 
term memory. 

3. Long-Term Memory 

Anderson (2000), Wickens (1992), and Dix et al. (1997) show sensory 
information is acted upon and processed by progressively higher neural centers, 
and as it continues to be acted upon, it will eventually be perceived or 
recognized, and then stored in long-term memory. Dix et al. (1997) contend 
long-term memory is a human beings main resource for storing factual 
information, experimental knowledge, and procedural rules of behavior (p. 30). 
Dix et al. (1997) discuss two types of long-term memory. There is episodic 
memory, which is a memory of events and experiences in a serial form. There is 
also semantic memory, which is structured to allow access to information, 
representation of relationships between pieces of information, and inference (p. 
31). Long-term memory differs from short-term memory in two very distinct 
ways. First, Long-term memory has a nearly infinite capacity. Second, forgetting 
happens at a much slower rate. 

4. Semantic Memory Maps 

Anderson (2000) states that in one’s semantic memory, information is 
broken down categorically. Thus, structure is represented as a hierarchy of 
categorized facts. Categorized facts are then associated with the different 
categories (p. 152). A semantic network is a series of interconnected semantic 
maps. Figure 2 shows a representation of a semantic memory map. 
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Schemas are developed from semantic networks. Anderson (2000) 
contends that since schemas are built from semantic networks, the sum of our 
knowledge about a semantic category can be captured (p. 154). Anderson 
(2000) also argues that categories have attributes, and that many of those 
attributes have a “default” value associated with them. Furthermore, if somebody 
recognizes an object as belonging to a certain category, that person can infer 
that the object has the default values associated with that concept’s schema (p. 
154). Figure 2 is an example of a semantic map. 
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Figure 2. A semantic mapping of ships (After Anderson’s (2000) model (p. 152)) 

5. Memory Relevance 

It is very important that curriculum developers understand the different 
aspects of human memory processes so CBT presentations can be constructed 
that take full advantage of the way people learn, and maximize the educational 
benefit to students. If a series of CBT lessons are related, and the ways in which 
they are related are incorporated into the presentations, the students are in fact 
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given the means to make mental connections between the various information 
groups and categories. Consequently, it is possible to use CBT to improve the 
way students construct semantic maps, ultimately improving memory and 
performance. 

B. KNOWLEDGE APPLICATION AND REASON 

Anderson (2000) describes three ways humans acquire knowledge. 
These are by personal discovery, through instruction (being told), and by 
observing somebody else’s performance (p. 246). 

1. Knowledge 

There are two types of knowledge. Anderson (2000) states these are 
declarative knowledge and procedural knowledge. Declarative knowledge is also 
known as explicit knowledge, and is the knowledge we are consciously aware of, 
including facts, dates, events, etc. Procedural knowledge is the knowledge of 
how to perform cognitive activities. This type of knowledge is often implicit (p. 
238). 

Knowledge acquisition, or learning, can be classified into two categories: 
tactical learning and strategic learning. As students practice problems they learn 
the sequences of actions required to solve problems or portions of problems. 
Anderson (2000) classifies this as tactical learning (p. 290). Students also learn 
how to organize problem-solving techniques, particularly as problems get large 
and complex. Anderson (2000) labels this as strategic learning (p. 292). The 
more information students learn about a subject, the more it is assumed they 
understand about that subject. Wiggins (1998) defines understanding as a 
sufficient grasp of concepts, principles, or skills so that one may bring them to 
bear on new problems and situations, deciding in which ways one’s present 
competencies can suffice and in which ways one may require news skills or 
knowledge (p. 84). Student understanding, then, must be a major goal of any 
educational endeavor. 

2. Reasoning 

Reasoning is the application of knowledge to solve problems or perform 
tasks. Dix et al. (1997) contend that reasoning occurs in three distinct ways: 
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deductively, inductively, and through abduction. Deductive reasoning means 
conclusions are logically derived from a set of given or known premises. 
However, a logical conclusion cannot be equated to “truth”. Just because 
something should logically be a certain way, does not mean it is really that way. 
When using inductive reasoning, a person generalizes from a set of known facts 
to infer information about cases that have not ^t been seen. Using abductive 
logic, a person would step logically from a given fact to the action or state that 
caused it (pp. 38-40). 

3. Expertise 

Over time, and through the use of learning strategies, reasoning, and 
problem solving, a person may become known as an expert about a given 
subject. Anderson (2000) describes three stages of skill acquisition and 
expertise. These stages are the cognitive stage, the associative stage, and the 
autonomous stage. In the cognitive stage, sets of relevant facts are committed to 
memory. In the associative stage, two actions occur. First, errors in the initial 
understanding are gradually detected and eliminated. Second, the connections 
among the various elements required for successful performance are 
strengthened (p. 281). The outcome from this stage is the internalization of a 
successful procedure for the performance of a given skill set. In the autonomous 
stage, Anderson (2000) states that procedural knowledge produced in the 
associative stage becomes more and more automated and rapid. Since skills 
are becoming more automatic, they require fewer processing resources (p. 282). 
For a given skill set, a person operating in this stage will generally be accepted 
as a subject matter expert (SME). 

C. TRANSFER 

Allessi and Trollip (1991) and Ellis (1965) define transfer as skills or 
performance in one situation influencing behavior in other situations. Transfer is 
also commonly used to describe taking information learned in an educational 
environment and applying it to situations that occur outside of the classroom. 
Ellis (1965) reports that the influence of transfer is found in intellectual tasks, 
complex motor skills, and in emotional responses and attitudes (p. 3). Three 
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types of transfer can occur as a result of instruction. They are positive transfer, 
negative transfer, and zero transfer. 

Positive transfer can be defined as the performance of one task aiding in 
the performance of a second task (Ellis, 1995, p. 4). Ellis (1995) argues the act 
of driving a car can be used to explain transfer. Assume a teenager has learned 
to drive the family car. Assuming a similar transmission, that teenager should 
have little difficulty driving the neighbor’s car (p. 4). 

Ellis (1965) defines negative transfer as the performance on one task 
inhibiting or disrupting the performance on a second task (p. 4). Ellis (1995) 
again uses the act of driving a car as an example. Learning to drive in the United 
States is different than learning to drive in Great Britain. A person that has lived 
and driven in the United States for a prolonged period of time would more than 
likely find it very difficult to drive in Great Britain, on the left side of the road (p. 5). 
This is an example of negative transfer. 

Zero transfer means that the performance of one task has no effect on the 
performance of another. 

There are many factors that affect the degree of transfer that occurs 
between one task and another. Ellis (1965) contends that task similarity, 
response similarity, and the time between tasks all influence how much transfer 
occurs. 

In general, increased task similarity results in increased positive transfer 
unless the stimulus is the similar but the responses are different. Ellis (1965) 
explains this phenomenon by demonstrating a motorist’s behavior while at a 
traffic signal. Consider changing the rules to “go” on orange instead of going on 
green. Since the color orange is similar to red, which means stop, motorists 
would have problems adapting to the new rules. However, if the green lights 
were replaced with blue ones, and then the rule was changed to go on blue, it 
would take little time before all the motorists had adapted to the change. 


11 



When a person must make new responses to similar stimuli, negative 
transfer usually results. Ellis (1965) uses a person learning another language as 
an example. Consider the person fluent in English who is attempting to learn 
German. Now consider the word, “wand”. In English, a wand is the Fairy 
Godmother’s tool. In German, it means a wall. Ellis’ research shows that the 
person will have a difficult time remembering and using the German definition 
properly. 

It was reported by Ellis (1965) that transfer remains constant, even with 
intervals that vary between zero and ninety days between tasks (p. 39). Transfer 
of learning appears to be stable and independent of any memory of the original 
tasks. 

Bastiaens and Martens (2000) contend that teaching students outside the 
context of the actual work environment results in a low transfer of knowledge and 
skill (p. 8). Not surprisingly, Bastiaens and Martens (2000) advocate the use of 
on-the-job training (OJT) for organizational training. It is possible that OJT has a 
few advantages over the traditional classroom approach. Bastiaens and Martens 
(2000) argue that in an OJT environment, the learning environment and the job 
location are the same, and positive transfer will increase (p. 8). It is also argued 
that since the learning process in an OJT environment is more active and 
concrete than in the classroom, transfer will improve (Bastiaens and Martens, 

2000, p. 8). 

While OJT does possess educational benefits, it is also likely that learning 
and transfer from this form of training are at times reduced. Transfer also occurs 
in the affective domain. The affective domain is primarily constructed of attitudes 
and opinions. A person that thinks poorly of his or her place of employment may 
possess negative feelings toward any OJT. It is also possible that the training 
day is often interrupted, which will diminish the impact of OJT. 

D. LEARNING PRINCIPLES 

The subject of learning is both broad and extremely complex. In order to 
construct a useful CBT program, it is important to have an elementary 
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understanding of how students learn and what instructors, even if the instructor is 
a computer, can do to enhance the learning process. Understanding the three 
learning domains that exist in each person, the concept of metacognition, and the 
four human factors that affect learning are very important. By understanding how 
these principles work together, curriculum developers can construct CBT 
software that will allow students to maximize their learning potential. 

1. Krathwohl, Bloom, and Bertram’s Domains 

Krathwohl, Bloom, and Bertram (1964) define three domains in which 
learning occurs. These are the cognitive, affective, and psychomotor domains. 
These domains are defined in terms of intellectual, mental, or physical objectives. 
Depending upon what a organization’s particular training objectives are, activities 
can be developed that stimulate student learning across each of these domains. 
If, however, an organization has poorly defined learning objectives, it makes it 
harder to develop specific training activities that will both meet the objectives and 
enhance learning and transfer throughout these three domains. 

a. Cognitive Domain 

This is the intellectual part of a person. This is the domain that is 
responsible for knowledge acquisition, recall, and problem solving. Krathwohl, 
Bloom, and Bertram (1964) define the cognitive domain as objectives which 
emphasize remembering or reproducing something which has presumably been 
learned, as well as objectives which involve the solving of some intellective task 
for which the individual has to determine the essential problem and then reorder 
given material or combine it with ideas, methods, or procedures previously 
learned (p. 6). 

b. Affective Domain 

Krathwohl, Bloom, and Bertram (1964) define the affective domain 
as objectives that emphasize a feeling, tone, emotion, or a degree of acceptance 
or rejection. This is simply a person’s attitudes, values, feelings, emotions, and 
biases (p. 6). 
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c. Psychomotor Domain 

Krathwohl, Bloom, and Bertram (1964) define the psychomotor 
domain as objectives that emphasize some muscular or motor skill, some 
manipulation of material and objects, or some act that requires neuromuscular 
coordination, such as writing, speech, or a trade (p. 7). 

2. Metacognition 

Stolovitch (2003) defines metacognition as the set of higher-level control 
processes that guide our deliberate information processing activities. It is in 
essence, the mind’s “Operating System” (OS) (p. 88). Stolovitch (2003) 
identifies five such processes. They are planning, selecting, connecting, tuning, 
and monitoring (p. 90). 

These processes are bipolar. On one end are characteristics that a good 
learner typically exhibits. On the opposite pole are the characteristics that a poor 
learner generally displays. Positions on this scale are not static, but dynamic. 
Over lime, a person can improve their performance within one or all of these 
processes’ spectrums with practice. 

a. Planning 

The process of planning ranges from a poor learner not knowing 
what to do to a good learner reasoning out what must be done by creating and 
executing an organized plan of action (Stolovitch, 2003, p. 90). 

b. Selecting 

The process of selecting ranges from a poor learner not knowing 
where to turn or select what is important to learn to the good learner listening, 
studying, analyzing, and sifting through information, identifying and selecting the 
critical elements (Stolovitch, 2003, p. 90). In other words, the good students are 
able to figure out what is important to learn. 

c. Connecting 

Stolovitch (2003) contends that those who do not connect well view 
new content as a mass to be analyzed and will attempt to memorize that content 
without linkages to known skills and knowledge. Those who do connect well 
continually seek to build linkages with their prior knowledge (p. 90). 
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d. Tuning 

Stolovitch (2003) argues that those that are poor at tuning will only 
obtain a fuzzy understanding of new knowledge and will be unable to pull that 
knowledge into focus. On the other hand, good students intentionally practice 
with new information until it is brought into a clear and sharp focus (p. 90). 

e. Monitoring 

Students with poor monitoring skills use known learning strategies 
whether those strategies work or not. These students tend to apply more effort 
into one strategy if a learning outcome is negative, or unproductive, than to apply 
a different strategy. Students who can effectively monitor replace unproductive 
or insufficient strategies with ones more likely to work (Stolovitch, 2003, p. 90). 

Stolovitch (2003) believes that teachers can help students of all 
ages improve in each of these areas by designing learning activities that guide 
students in the application of these skills, which forces them to actively think 
about and evaluate what they are doing and why they are doing it (p. 88). 

3. Factors that Affect Learning 

There are four factors that have an impact on the way a student learns. 
These factors are: ability, prior knowledge, motivation, and thinking style. 

a. Abiiity 

Stolovich (2003) defines ability as the capacity with which we were 
born that enables us to acquire new skills and knowledge (p. 36). This is also 
known as intellectual capacity, and it varies from person to person. Those with a 
high intellectual capacity will typically understand and recall information better 
than those without such a capacity. 

b. Prior Knowiedge 

Prior knowledge is defined as how much a person already knows 
about what is being currently taught. Stolovitch (2003) states that prior 
knowledge helps the learner acquire additional knowledge or skills more quickly 
(p. 37). 
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c. Motivation 

Motivation can be defined several ways. I prefer to define 
motivation as a person’s desire to achieve a stated purpose or goal. Stolovitch 
(2003) believes that motivation is affected by three factors: value, confidence, 
and mood (p. 37). Stolovitch (2003) argues, and it makes sense, the more a 
person values something; the more motivated that person will be with respect to 
what is valued. In terms of learning, this means that the more value a person 
affixes to learning, the more motivated that person will be to learn. 

Additionally, Stolovitch (2003) argues that confidence plays a large 
part in a person’s motivation. If a person feels completely inept at something it is 
unlikely they will be motivated to do or try it (p. 38). 

Stolovitch (2003) believe that a person’s personal feelings affect 
their moods, and that those moods affect motivation (p. 38). A positive 
environment tends to improve one’s mood, and consequently, improve motivation 
(Stolovitch, 2003, p. 38). 

There are two forms of motivation. Allessi and Trollip (1991) 
describe them as intrinsic motivation and extrinsic motivation. Intrinsic 
motivators are things that are inherent in the instruction that motivate a student 
(p. 31). Allessi and Trollip (1991) describe exploratory environments, challenging 
assignments, tasks that stimulate curiosity, and encouragement as methods that 
increase a student’s intrinsic motivation (p. 31). Allessi and Trollip (1991), 
describe extrinsic motivators as those that are independent of the instruction (p. 
31). Paying a student for good grades is an example of an extrinsic motivator. 
Allessi and Trollip (1991) argue that when extrinsic motivators are used, the 
student’s interest is diminished because the goal becomes the reward instead of 
learning (p. 31). 

1. Learning Environment and Motivation. The learning 
environment can have a large impact on a student’s motivation. There are four 
different learning environments. Kirkpatrick (1998) describes them as 
discouraging, neutral, encouraging, and requiring. In a discouraging 


16 



environment, the supervisor often says, “I want it done this way”, irrespective of 
the actual organizational rules. A person is not often told directly not to do 
something, but behavior is regulated through insinuation. According to Allessi 
and Tollip (1991), a neutral environment is one in which the boss ignores the fact 
that trainees have attended training (p. 21). In effect, the boss is saying, “I do not 
care what you have learned, do it this way.” In an encouraging environment, the 
organization assists the student in transferring what was learned in training to the 
job. The “requiring” environment is the most difficult to implement. In this 
environment, the supervisor knows what the subordinate learns and then 
ensures that the students’ learning transfers to the job (Allessi and Trollip, 1991, 

p. 21). 

Based upon these descriptions, organizations should strive 
to implement either encouraging or requiring environments. By doing so, more 
transfer will take place between training and the job, trainee satisfaction and 
motivation will increase, and the organization will benefit from a better-trained, 
more responsive workforce. 

d. Thinking Styies 

Sternberg (1997) argues that in addition to ability, prior learning, 
and motivation, thinking styles significantly impact a person’s ability to learn. 
Sternberg (1997) identifies three distinct thinking styles. They are legislative, 
executive, and judicial. Legislative thinkers like to come up with their own ways 
of doing things. They also prefer to decide for themselves what they will do and 
how they will do it by creating their own rules. Legislative thinkers tend to make 
good writers, scientists, artists, and bankers (Sternberg, 1997, pp 20-21). 

Sternberg (1997) states that executive thinkers like to follow rules 
and prefer problems that are prefabricated. They like to fill in gaps that exist 
within existing structures rather than create the structures themselves. They also 
like applying rules to problems. These types of thinkers make good lawyers, 
police officers, contractors, and soldiers (p. 21). 
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Judicial thinkers like to evaluate rules and procedures, preferring 
problems in which one analyzes and evaluates existing constructs and ideas 
(Sternberg, 1997, p. 21). Sternberg (1997) argues that these people like writing 
critiques, giving opinions, and judging people and their work. Judicial thinkers 
make good judges, critics, program evaluators, consultants, and systems 
analysts (p. 21). 

Since people all operate using different thinking styles, questions 
placed on a test will usually result in different answers, depending on the thinking 
style of the person asked. It is important to keep each of these styles in mind 
when constructing tests and activities with which to evaluate students. This is 
because a test constructed one way may favor a particular thinking style over 
another. Asking for the same information in different ways or constructing 
multiple activities that demonstrate the same skill are methods for eliminating any 
bias that may exist with respect to the students’ thinking styles. 

E. APPROACHES TO LEARNING 

There are many theories about how to best teach students. The three that 
will be discussed in this research are the behaviorist theory, the constructivist 
theory, and the cognitive theory. Some theories have more relevance in a 
traditional classroom, while some seem to fit best within the confines of some 
computer-based environments. Each theory has both advantages and 
disadvantages with respect to the learning environment in which it is applied. 
The goal of this research with respect to learning theories is to highlight the basic 
similarities and differences between the theories and to briefly discuss possible 
strengths or weaknesses of the three theories in the context of organizational 
training via CBT. 

1. Behaviorism 

Smith-Grato (2000) traces the roots of behaviorism to Skinner’s Theory of 
Programmed Instruction (p. 228). Programmed instruction was founded on 
learning principles largely determined by animal learning studies. 

Kidd (1965) believes that learning is primarily the alteration of behavior 

that results from experience (p. 154). Kidd (1965) is saying that we learn from 
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what we do, not from what we accomplish (p. 154). When I was a young boy, I 
touched a hot burner on a stove. Having learned from my experience, I have 
never again touched a stove burner that may even have been hot. 

Kidd (1965) describes the principle known as the “curves of practice”. 
This principle demonstrates that for a given activity, the gains in learning or 
expertise increases the more that activity is practiced (p. 156). Consider a 
person learning to play the guitar. The more a person practices playing the 
guitar, the better he or she will become. 

Desypris (2002) lists three steps that behaviorists believe need to be 
performed in order to effectively teach. They are: 

(1) Instructors must identify the objectives to be learned. 

(2) Instructors must create a learning environment that assists the learner in 
acquiring these goals. This environment includes stimulus that will engage the 
learner. 

(3) Instructors must review, examine, and consider adopting existing materials 
into their curriculums before developing new ones (p. 18). 

Behaviorism was a very popular theory in the 1950’s and 1960’s. 
Behavioral learning principles tended to emphasize simple, incremented steps 
representing a particular task domain. The behavioral method of programmed 
instruction has been successfully applied in limited training environments on 
tasks that require performance of simple procedures. Even though this form of 
instruction largely ignores human problem solving abilities and higher cognitive 
skills, it still provides a nice foundation on which to build. 

2. Cognitive Theory 

Duffy and Jonassen (1992) describe cognitive science as multidisciplinary, 
drawing on psychology, linguistics, anthropology, philosophy, and artificial 
intelligence (p. 20). Duffy and Jonassen (1992) argue that the mind is essentially 
viewed as an instantiation of a computer, manipulating symbols the same way 
that a computer does. These “symbols” acquire meaning when an external and 
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independent realty is “mapped” on them as during our interaction with the world. 
Cognition is the rule-based manipulation of these symbols (p. 20). Leflore (2000) 
states that individuals develop maps and schemas to help them understand the 
world. Individuals will reorganize existing concepts when those concepts interact 
with new experiences (p. 105). 

Because of this, Desypris (2002) argues that the course must include 
problems that the student will solve in step, taking advantage of acquired 
knowledge in each step (p. 18). Additionally, Desypris (2002) contends that in 
order for instruction to be effective, it is the teacher’s responsibility to ensure that 
each student is ready to learn the core concepts that will be taught. 

Leflore (2000) provides four basic guidelines that instructors can utilize 
when developing course content: 

(1) Provide students with elements that help them structure and organize the 
information they are expected to learn. Providing students with an outline of the 
material they are going to learn is an excellent method for accomplishing this 
task (p. 105). 

(2) When appropriate, use a concept development activity (p. 105). An example 
of this would be to ask aspiring students in a nautical navigation class to draw the 
light configuration of a particular ship, or class of ships. 

(3) Decide how students’ prior knowledge will be activated (p. 105). 

(4) Use graphics, animations, and sounds that are related to the content being 
taught (p. 105). 

3. Constructivism 

The theory of constructivism has gained in popularity as the Internet, and 
distance learning has gained in popularity. Duffy and Jonassen (1992), Leflore 
(2000), and Smith-Gratto (2000) contend what makes this theory unique is its 
tenet that the students construct their own meaning, or schemas, based on their 
unique individual experiences. Duffy and Jonassen (1992) argue learning is an 
active process, built on the internal representations of knowledge, which has 
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been constructed from personal interpretations of one’s experience. This 
knowledge is constantly open to change and its structures and linkages form the 
foundation to which knowledge structures are appended (p. 21). One way of 
interpreting this viewpoint is through the semantic mapping model. When a 
person is young, the structure of their semantic map is limited and small. Over 
time, as this person experiences life, those mappings become large, complex, 
and interconnected. 

Duffy and Jonassen (1992) and Leflore (2000) contend that for any 
meaningful learning to occur, it must be situated in a rich context and filled with 
real-world problems for students to solve. Leflore (2000) states simulations 
provide enough fidelity for students to explore real-world problems from multiple 
vantage points and will enable students to “construct” good problem solving skills 
(P-112). 

According to Desypris (2002), this theory puts students in charge of their 
own learning. Consequently, course content must be well structured in order to 
provide continuity of learning to the students. Since experience plays such a 
large role in this theory, interaction among participants is a very important 
element in the construction of structured knowledge. 

Smith-Gratto (2000) describes three potential problems that can occur 
when teaching using this method: 

(1) Students may not meet the required objectives because facts, skills, and 
concepts may not have been learned. This can happen if students are required 
to make meaning from unfamiliar content. If students’ experiences do not 
overlap with new information, it is unlikely that the concept will be successfully 
learned (p. 233). 

(2) It is possible that students will not construct a proper foundation on which to 
build (p. 233). 

(3) Constructivist activities are very time consuming (p. 233). 
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4. Conclusion 

By understanding how the human mind receives, stores, processes, and 
retrieves information, CBT developers can design courseware solutions that give 
students the best opportunity to maximize their learning potential. On the other 
hand, it is likely that CBT courseware that is neglectful of how we learn will fail to 
meet the organizational objectives for which it was designed. 

Although these three theories have been presented independently, there 
is nothing to prevent curriculum developers to combining ideas and activities 
from each one in order to provide an optimum learning environment for their 
students. By doing so, developers can maximize the benefit offered by a 
particular theory and minimize the associated weaknesses that are also inherent. 
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III. COMPUTER-BASED LEARNING 


For any educational initiative to be successful, whether it is teaching 
English to fifth-graders or training potential pilots how to master the mechanics of 
a particular aircraft, two things must occur. First, the curriculum must be sound. 
This means, among other things, the information being presented must be 
accurate, comprehensive, relevant, and timely. Second, the information must be 
effectively communicated, or delivered, to the students. Using appropriate 
Instructional Design techniques generally allows for the development of a sound 
curriculum. Proper pedagogical principles must be adhered to in order to 
produce quality instructional presentations. 

The quality of instructional design and delivery techniques is assessed 
through the use of evaluations. Evaluations allow organizations to determine if 
training, and the performance goals that arise out of the completion of the 
training, are being met or exceeded. The evaluation framework that is presented 
at the end of this chapter was constructed by researching instructional design 
and pedagogical principles that have been proven to be effective in a CBT 
environment and then building a template that checks to see if those 
characteristics are present in the course of instruction being evaluated. 

A. INSTRUCTIONAL DESIGN 

Instructional Design can be defined as, “The process of designing the 
environment, methods, and resources for effective learning and objectives by 
students (Boettcher and Conrad, 1999, p. 42). As quoted by Desypris (2002), 
Fliltz, Teles, and Turoff contend, “attention to instructional design is one of the 
most critical factors in successful learning networks... All education involves 
intervention by an expert to organize the content, sequence instructional 
activities, structure task and group interaction, and evaluate the process (p. 20).” 

Bloom (1956) describes curriculum development, or instructional design, 
as a four-step process that begins with asking what educational purposes or 
objectives the course seeks to attain? Once answered, the developer must 
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create learning experiences which will likely bring about the attainment of the 
objectives. Next, the course should be sequenced to enhance transfer and avoid 
isolated learning experiences. Finally, the effectiveness of the learning 
experiences must be evaluated (p. 18). In their faculty guide, Boetcher and 
Conrad (1999) develop a comprehensive four-part model for developing 
instruction that does not conflict with Bloom’s early model. The model’s 
components are comprised of Analysis, Learning Objectives, Selection, and 
Evaluation (Boetcher and Conrad, 1999, p.17). 

1. Analysis 

The first step in the instructional design process is to conduct an analysis 
of the students and what they will be learning. A teacher must know who the 
students are, what they already know or do not know, and what they will need to 
know to consider the training program a success. 

In his book Educative Assessment, Wiggens (1998) defines a standard as 
something that describes a specific and desirable level or degree of exemplary 
performance irrespective of whether most people can or cannot meet it (p.106). 
He also contends that for any education or training program to achieve success, 
three standards must exist. First, Content Standards need to be in place. This 
describes what the student should know and be able to do (p. 106). Second, 
Performance Standards need to be created. This standard describes how well 
students must do their work. Third, Task Standards must be implemented. This 
describes what tasks the students must be able to perform (p. 106). 

It is important that designers realize that without analyzing what the 
training program is intended to accomplish, what the students need to know, and 
how to best teach the students what you want them to learn, the training initiative 
may very well fail. 

2. Learning Objectives 

The learning objectives become the standard by which the training 
program is judged. Poorly defined learning objectives hinder the evaluation 
process, of both the students and the program. Learning objectives formally 


24 



define what it is the students are expected to learn as a result of participating in 
the training program. 

3. Selection 

Selection describes the process of choosing teaching strategies, picking 
the content that will be taught, and selecting the methods that will be used to 
assess the program. More than one teaching strategy may be appropriate for 
teaching a particular class. For example, an instructor may decide to use 
multiple teaching strategies by using Internet delivery methods coupled with 
several classroom lectures. Content selection is very important. The instructor 
should verify that the course content would satisfy the learning objectives. 
Methods used to assess both the course and the students should be selected. 
This means the developer must decide what is to be evaluated, how it is to be 
evaluated, and when it will be evaluated. Once this is decided, the developer 
must decide what to do with the information collected via the evaluations. 

4. Evaluation 

There are two fundamental evaluation types: Those performed by the 
instructor, and those performed by the institution. Instructors generally evaluate 
student performance through observation, quizzes, and examinations. 
Institutions usually assess the curriculum, the instructor, or the program. Opinion 
surveys, questionnaires, and direct observation by organizational representatives 
are common methods of evaluation employed by institutions when evaluating 
program effectiveness. 

B. PEDAGOGICAL DESIGN 

All educational curriculums must be properly delivered if their content is to 
be understood, remembered, and used later by the students. Even the best 
instructional designs can be rendered impotent if the content is not efficiently and 
effectively presented to the students. It is not difficult to envision a teacher 
presenting otherwise interesting material to students in a way that is so boring 
most students in the class are uniformly wondering when the lesson will be 
mercifully over (how good the instructional design was becomes irrelevant if and 
when this ever occurs). A poor teacher can easily render an entire course 


25 



ineffectual through poorly designed, or if designed well, poorly executed delivery 
methods. To be successful, different delivery methods must employ different 
tactics to gain the students’ attention, peak their interest, and motivate them to 
learn the material. The traditional classroom environment emphasizes different 
techniques than does the computer-based environment, although the ultimate 
goal is the same. 

1. Classroom Instruction versus Computer-based Delivery 

Traditional instruction and computer-based training both have advantages 
and disadvantages. A course developer must be familiar with the pros and cons 
of each technique in order to select a delivery method that best suites the 
organization’s needs. The traditional classroom environment features 
information taught by an instructor or through a textbook. This information is 
taught linearly, or in a set sequence. Classroom training is synchronous, 
meaning that communications take place in real-time. Web-based and computer 
based training can provide information from varied sources, many times via a 
multimedia presentation that makes use of hyperlinks. A hyperlink is an 
embedded doorway that when clicked on with the mouse pointer, takes the 
student to the webpage associated with that link. Hypermedia is the use of 
multiple hyperlinks, and through the use of those links, allow the students to 
control where they go within the framework of the lesson in order to obtain their 
learning material. Multimedia is the use of two or more forms of communication 
(e.g. pictures, sounds, video, etc) to present information to a target audience via 
a computer. Communication with the instructor can be asynchronous or 
synchronous. Asynchronous means that there is a delay between the time a 
communication is sent and the time it is received. Electronic mail (email) is an 
example of asynchronous communication. Table 1 highlights the differences 
between traditional training and web-based training methods. 
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Lecture-Textbook 
Learning Paradigm 

Web-based Learning 
Paradigm 

Main Sources of 
Information 

Teacher or Textbook 

Various Internet Resources 
including online textbooks 

Format of Information 

Mostly Lecture or Text 

Mostly Multimedia 

Presentation Format 

Linear 

Mostly Hypermedia 

Interaction Type 

Synchronous 

Asynchronous/Synchronous 


Table 1. Differences between Lecture-textbook and web-based learning paradigms 

Horton (2000) describes several advantages to instructor-led training: 

(a) The instructor can answer questions and solve problems as they arise (p. 55). 

(b) Instructors provide authority that some learners may need for motivation (p. 
55). 

(c) An instructor can adjust the course to suit the needs of a particular class (p. 
55). 

(d) Instructors can grade the activities and tests too subtle for automated scoring 
(p. 55). 

In my opinion, the ability of instructors to motivate the students to learn is 
the most important of these characteristics. Since an unmotivated student is 
likely to learn less than a motivated student, it is very important that instructors 
are able to effectively perform this function. This does not mean an instructor 
needs to take on the role of an enforcer, as the connotation of motivator might 
imply. It does mean that the teacher is able to provide a rich, interesting, and 
safe learning environment that attracts and inspires students to learn. 

Traditional classroom instruction also possesses a few inherent problems. 
First, not all instructors are created equal. Multiple instructors may teach the 
same course of instruction, which means students taking the same class may not 
receive the same quality of instruction. Second, this is generally considered 
“passive” learning, meaning the teacher stands in front of the class and lectures 
while the students take notes. This form of learning is not very interactive, which 
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does not allow the students much time to mentally work with the information and 
transfer it from STSS to long-term via short-term memory. Consequently, This 
style of teaching may not produce as much positive transfer among the student 
population as other more interactive methods. 

Computer-based training possesses several advantages over traditional 
instruction. In Designing Web-Based Training, Horton (2000) lists several: 

(a) Learners develop self-reliance that is needed after the class (p. 55). 

(b) Learners are not required to conform to the instructor’s schedule (p. 55). 

(c) All learners get the same quality of learning experience (p. 55). 

Developers today can create stunning graphics and visual effects, embed 
video and audio, and produce highly entertaining and interesting presentations 
that grab and hold students’ attention. Coupled with the advantages listed above 
by Horton, powerful learning tools can be created that allows the students to, in 
essence, take control over their own learning. 

Despite its potential powerful advantages, CBT does have several 
possible disadvantages. One of weaknesses of computer-based training is that it 
is generally more expensive to deliver than traditional teaching methods. In 
today’s fiscally conscious environment, it is not surprising that most courses and 
CBT web sites are not very interactive, and fall well short of initial expectations 
(Parikh and Sameer, 2002, p. 28). 

Another potential drawback of CBT is that there is substantially less 
instructor control and struggling students may not get the much-needed help they 
require to successfully complete the training. If students feel isolated, or are not 
motivated to learn, they may not take responsibility for their coursework. 
Subsequently, this may lead to a high rate of attrition. Unmotivated trainees are 
unlikely to seek help if cumbersome processes are involved. 

Parikh and Sameer (2002) state there is no feedback loop between 
instructors and students unless a technically complex and cumbersome login 
function is added to the courseware (p. 29). This type of secure instructor- 
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student feedback loop can add considerable costs to a CBT or web-based 
training initiative. However, not including this functionality with the CBT 
courseware may lead to feelings of isolation and the subsequent motivation 
issues discussed in the previous paragraph. 

Finally, there is the potential to rely too heavily on technology when 
designing CBT. In the end, technology cannot replace the human teacher in 
education. Yeung (2002) believes that although technology can be used to 
leverage instructors’ time, it cannot replace most human contact without 
significant quality losses. 

2. Stolovich’s Five-Step Model for Training 

It is essential to formally structure the training students receive. This is 
true regardless of the delivery method, as it reasonably assures the organization 
that all students will receive approximately the same training irrespective of who 
the instructor is. Stolovitch and Keeps (2003) present a comprehensive five-step 
model for structuring training. Figure 3 shows this model. This model includes 
the following steps: 

(1) Rationale. Explain what it is you want the students to learn. This serves to 
motivate the students to learn the important information since it has a high value 
associated with it (p. 68). 

(2) Objective. Stolovitch and Keeps (2003) suggest that by telling students what 
they will be able to do at the end of the lesson, there is a better chance they will 
learn it (p. 70). 

(3) Activities. Learning activities guide students toward the fulfillment of the 
stated learning objectives (p. 71). These activities should be stimulating, 
contributing to students’ experiences, imaginations, and judgments (p. 71). 

(4) Evaluation. Student evaluation verifies what the students have learned. It 
determines the degree to which the learner has met each objective for the 
desired level of performance (p. 72). The most common evaluative techniques 
are tests and performance appraisals (p. 73). 
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(5) Feedback. Done correctly, feedback lets the learners know what they did 
correctly and provides constructive help when they have done something wrong 
(p. 73). There are two forms of feedback: corrective and confirming. Corrective 
feedback explains how students can obtain, or meet, a stated learning objective 
(p. 73). Confirming feedback is a reward for attaining a learning objective (p. 73). 



Figure 3. Five-step Model for Training (From Stolovitch and Keeps, 2003, p. 69) 

Although Stolovitch and Keeps (2003) present a fairly comprehensive 
model, it lacks a function that verifies the instructional interface is appropriate for 
the training being conducted. It is important to examine instructional interface in 
CBT because the computer becomes the teacher. Flow a student interacts with 
the computer during CBT many have an impact on whether the student is able to 
realize the maximum educational benefits of the training. The evaluation 
framework presented at the end of this chapter will assess instructional interface 
in CBT. Lohr and Eikleberry (2001) define instructional interface as, “the 
elements in any product or system that support the tasks of the learner while he 
or she is learning (p. 25).” Lohr and Eikleberry (2001) continue by stating this 
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interface can be human-based, computer-based, paper-based, or a combination 
thereof (p.25). This interface is the link between the student and the information 
or tasks being learned, and appears to be directly linked to the teaching method 
and is an integral part of answering some of the fundamental questions students 
ask, such as, “Am I being graded?” or “What am I supposed to do now?” (Lohr 
and Eikleberry, 2001, p. 25). When a CBT system lacks human instructors that 
would otherwise guide, inspire and answer questions, a well-designed and 
properly functioning instructional interface is critical. Instructional Interface will be 
examined in more detail in a later section of this thesis. 

3. Computer-based Pedagogical Design Principles 

Distance Learning can be an efficient way for organizations to conduct 
training. Computer-based instruction using active, self-paced learning with 
individualized feedback can reduce total learning time by 30 to 40 percent 
(Desypris, 2002, p. 13). In fact, Horton reports that California State University at 
Northridge, Office Depot, Merrill Lynch, and Toys R Us have all generated 
significantly better training results since beginning Distance Learning initiatives 

(p. 26). 

a. Potential Drawbacks 

Despite these successes enjoyed by many organizations, CBT is 
not without potential problems. It is possible that meanings will be misinterpreted 
(Horton, 2000, p.37). If a student does not adequately understand the instruction 
then it is likely that some of the presentation will be misinterpreted. The fact that 
there is no human instructor only exacerbates this problem if it occurs. Another 
potential problem that arises from the use of CBT is that there are very few ways 
to get a student “unstuck” (Horton, 2000, p. 37). Horton (2000) also raises the 
fact that there is very little socialization that occurs when one learns alone (p. 
37). He argues that this leads to two potential problems. First, a student is 
unable to learn from his or her classmates (p. 37). Second, if lonely because of 
insufficient socialization, a student’s motivation will decline (p. 37). Lastly, there 
are too many possible disruptions (p. 37). As one who has first-hand experience 
with classes that have had much of the required information presented on-line, I 
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can attest to the fact it was often difficult to find, download, read, and absorb the 
material in a timely manner due to all the interruptions and disruptions. In my 
opinion, this is potentially the biggest problem area of all. Unfortunately, it is also 
the one sphere of influence that a course developer has the least amount of 
control over. 

b. General Issues 

Most of CBT’s potential pitfalls can be avoided by taking advantage 
of the development principles others have found successful. As a general rule, 
Allessi and Trollip (1991) contend that the course developers must answer 
several questions during the development process: 

(a) Is the use of the computer appropriate (p.486)? 

(b) Is the methodology appropriate (p. 486)? 

(c) Are the lesson lengths appropriate (p. 486)7 

(d) Is the mastery level appropriate for each topic (p. 486)7 

c. Student Interaction 

Once these questions are satisfactorily answered, developers can 
critically examine other aspects of their initial design. How the students will 
interact with the computer is an issue developers must closely scrutinize. 
Belanger and Jordon (2000) contend that interactivity is the key element of the 
learning environment, and define it as the degree of intellectual, emotional, and 
physical engagement of the learner to the instructional content using computer- 
based Distance Learning technologies (p. 23). Belanger and Jordon (2000) 
describe four successively more complex levels of student interactivity. They are 
from least to most complex: Passive, Limited Participation, Complex 
Participation, and Real-time participation (p. 134). Passively oriented text and 
graphics are not generally considered the optimal choice for use when 
developing an integrated CBT program. This method can be used for teaching 
sets of basic facts or rules (Belanger and Jordon, 2000, p. 134). 

Interactive text presents learners with different pages of content, 
depending on the selections they make during the course of the lessons. 
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Belanger and Jordon (2000) recommend using an index or hyperlinks for 
navigation (p. 138). By using hyperlinks throughout a lesson, information from 
one lesson can be linked to information in other lessons. Belanger and Jordon 
(2000) believe interactive text and graphics reinforces the ways in which the 
material relates to other topics. The amount of student transfer increases 
because facts and processes just learned are immediately practiced (p. 139). 

The most expensive, complex, and fully interactive type of distance 
learning instruction is interactive multimedia (Belanger and Jordon, 2000, 139). 
Belanger and Jordon (2000) explain that when using interactive multimedia, 
interactive text is combined with video, sound, animation, and high-resolution 
graphics to provide a rich learning environment (p. 139). According to Belanger 
and Jordon (2000), a highly interactive CBT environment decreases students’ 
sense of isolation, allows more senses to be used during the learning process, 
and students are engaged vice learning through passive listening, which 
increases interest and motivation (p. 21). 

It is important that developers properly select the most appropriate 
level of interactivity with respect to organizational training objectives. Different 
lessons, or even different parts of the same lesson, may require different levels 
of interactivity. These requirements, linked back to the learning objectives, are 
what drive the interactive complexity of each topic. 

Allessi and Trollip (1991) recommend that developers ask five basic 
questions about how students will interact with the computers when deciding on 
how interactive to make a lesson: 

(a) Is interaction frequent (p.486)? 

(b) Is comprehension enhanced (p. 486)? 

(c) Is memory enhanced (p. 486)7 

(d) Is transfer enhanced (p. 486)7 

(e) Are there a variety of student interactions with the computer (p. 486)7 
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If these questions cannot be answered in the affirmative, then it is 
probable that students will be victimized by at least one of the problems listed in 
this chapter, such as boredom, low motivation, misinterpretation of instructional 
material, inability to proceed any further, etc. Frequent interaction with 
interesting material keeps students’ interests high while simultaneously providing 
motivation to learn. From experience, a series of long PowerPoint presentations, 
where the students only interaction is hitting the enter key to advance along one 
slide at a time is neither interesting nor motivating. 

Another approach used to assess this aspect of design is to ask if 
there is a teacher in the interface (Lohr and Eikleberry, 2001, p. 24). This takes 
into account the fact a human instructor is absent while the student is actively 
learning. By frequently asking this question while designing a course, developers 
can build-in software applications that take on the role of an instructor, or an 
assistant, while students are using the program. 

d. Human-Computer Interface 

There are many different types of human-computer interaction 
styles which developers can implement for use in a training program. It is best if 
one style is selected and used throughout the entire range of associated training 
topics instead of changing styles from topic to topic. Changing styles will only 
confuse and frustrate the students taking the class. In fact, Florton (2000) argues 
that designers should modularize the course, separating it by loosely associated 
topics. This will minimize the dependence between topics (p. 174). Whichever 
method is used, the students will need to use an input device (e.g. mouse, 
keypad, etc) to interact with the computer. Allessi and Trollip (1991) contend that 
designers must carefully select the method of inputting selections into the 
computer and that the selected method’s interface should assist in preventing or 
detecting any student errors. Some of the more popular styles include Menus, 
WIMP (Windows, Icons, Menus, and Pointers), and Point and Click (Dix et al., 
1997, p. 115). 

Dix et al. (1997) describe menus as a set of on-screen options that 

are selected using an input device. For menus to be beneficial to the students, 
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menu options must be meaningful and logically grouped together (Dix et al., 
1997, 117). Horton (2000) lists three suggestions for making a menu driven 
interface more effective. First, build in an index and search function so learners 
have more ways to find related topics (p. 174). Second, display menus in 
separate frames. Learners can then navigate to related topics via the menu 
rather than by links within the main body of the web or course pages (p. 174). 
Third, automate sequential navigation. Develop the software so the “next” and 
the “previous” buttons calculate where to go based on an easily maintained list of 
topics (p. 174). 

Dix et al. (1997) portray the WIMP interface as an interface that 
contains windows, icons, menus, and pointers. It is similar to the Windows 
based operating system that is found on most of today’s personal computers. 

Point and click styles utilize a menu that students use a pointer 
(mouse) and click on the option that they desire to initialize (Dix et al., 1997, 
p.129). Horton’s suggestions regarding the use of menus, such as building an 
index and search function, using separate frames, and automating navigation are 
also appropriate to this interface style. 

e. Program Control Issues 

There is a wide array of non-instructional control issues that must 
be resolved prior to students participating in a computer delivered course of 
instruction. These issues include directions, book marking during a lesson, 
finishing a lesson, and moving forward and backward throughout a lesson. 
Methods available for students to use when activating one of these control 
functions can include menus, icons, or special keys on the keyboard that initiate 
specific functions. Allessi and Trollip (1991) recommend using menus for such 
things as initial control, sequence selection, and review functions (p. 26). Allessi 
and Trollip (1991) further advocate saving command key usage for forward 
progression, backward paging, help, index, and exit functions (p. 26). While 
Belanger and Jordan (2000) do not discriminate between menus and command 
keys, they argue that learners must be provided with the ability to add, delete. 
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and organize bookmarks; track progress; use a review function; and the ability to 
generate and use practice quizzes and tests (p. 149). 


Every student needs the capability to find and read the courseware 
directions. Allessi and Trollip (1991) claim directions are essential to the 
effectiveness of any computer-based lesson (p. 21). Whether directions are 
optional or mandatory, Allessi and Trollip argue that it is vital to ensure they are 
readily available (p. 21). This makes sense. If optional directions were not easy 
to find and use, students would quickly give up and the lesson would not be 
satisfactorily completed. A paper-based manual may be the best method for 
delivering directions to students. This manual can be saved on a server in an 
Adobe PDF format, and then downloaded via the Internet by students. Students 
will be able to more easily access and use directions in the event of software 
“glitches” if the directions are obtained separately and used independently from 
the courseware. 

From a student’s perspective, the ability to bookmark a spot in a 
lesson, leave, and then come back to it is important. Students may not have the 
time to successfully complete a given lesson in one sitting. No student wants to 
start a lesson, have to leave in the middle of that lesson, and then have to restart 
from the beginning again when they came back. This is a waste of time and 
effort. Consequently, the ability to bookmark a lesson is critical to students’ 
attitudes toward their learning environment, and quite possibly to their successful 
completion of a course of i nstruction. 

Allessi and Trollip (1991) recommend students be given the option 
to quit the program, return to the main menu, or continue to the next planned 
lesson whenever they complete a topic (p. 85). Empowering the students to 
decide what topic to complete next gives the students a sense of control, and 
improves motivation to continue using the software. 

Another vital control function students need is the ability to page 
forward and backward throughout a lesson. Sometimes a student is very familiar 
with, or has already mastered, the particulars of part of a topic. The student may 
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want to rapidly scan through the portion of material he or she has mastery of, and 
the incorporation of a page forward key or icon is a good way to give this control 
to the student. Alternately, if a student has not understood the material over the 
course of several slides in a topic, it is essential to give that student the ability to 
go back and reexamine those slides. In this way, a student can repeatedly go 
through any section of a lesson until it is understood or mastered. 

f. Design Evaluation 

Once an organization has developed an initial design for a training 
program, it is best to evaluate that design. Dix et al. (1997) offer many useful 
ideas about evaluating designs that can be useful in determining if the initial CBT 
blueprint is correct. He suggests the CBT system should always keep users 
informed about what is happening via timely feedback. Since consistency is key, 
developers should verify that users do not have to wonder whether different 
words, situations, or actions mean the same thing. The user should not have to 
remember information from one part of a dialog to another and instructions 
should always be available. Despite the prevalence of computers, there are still 
many people who have a difficult time operating them (p. 414). According to Dix 
et al. (1997), it is also important that all CBT systems be able to help users 
recognize, diagnose, and recover from errors (p. 414). By verifying a CBT 
program is designed to satisfactorily meet these criteria, developers can help 
ensure a relatively smooth transition from the design phase of a CBT initiative to 
all subsequent phases with minimal design rework. 

4. Lesson Organization and Sequencing 

If a training initiative is going to be effective, it must be efficiently 
organized and the lessons should be properly sequenced. In other words, the 
effort must be properly framed. Horton (2000) describes a course’s framework 
as the routine parts of the course that do not teach subject matter but are 
nevertheless an essential part of the course (page 78). Allessi and Trollip 
(1991), along with Horton (2000), agree that these can include, but are not 
limited to the following: Course announcements page, course and lesson 
introduction pages, learning objectives pages, learning materials, and a feedback 
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loop. Some authors, such as Allessi and Trollip (1991), include the course’s 
actual instructional content within a course’s framework. 

a. Announcements Page 

Although it may appear unnecessary to develop an announcements 
page if conducting training exclusively with CBT, it is still a good idea to 
implement. There are two basic reasons why. First, just because the training is 
conducted via the computer does not necessarily mean that testing is. An 
announcements page is a convenient method to announce quizzes, tests, and 
performance appraisals. Second, the instructional information on a particular 
compact disc may need to be changed. An announcements page on the Internet 
allows administrators to easily pass all relevant information regarding changes to 
the students. 

b. Introduction 

In CBT, the introductory part of a lesson consists of two parts. The 
first is the title page. The title page is used to attract attention, create a receptive 
attitude, and to indicate what the topic is about (Allessi and Trollip, 1991, p. 19). 
An unattractive, sloppy, or overly simplistic title page may give students an idea 
the presentation is going to be unattractive, sloppy, or boring, which will decrease 
students motivation to progress through the lesson before any learning has 
occurred. 

The second part of the introduction is the presentation of learning 
objectives. As indicated previously, learning objectives indicate what the student 
will be able to do, say, or write at the completion of the lesson (Allessi and Trollip, 
1991, p. 19). For example, a typical learning objective may look similar to, “After 
this lesson, you will be able to set the time on your VCR.” Allessi and Trollip 
(1991) identify two types of learning objectives, behavioral and non-behavioral. 
A behavioral objective expresses the task, conditions, and performance required 
by the student. Reporting contact with another ship to the Captain is an example 
of a task routinely performed at sea. The behavior objective for this task would 
include the prerequisite conditions that must exist prior to starting the procedure, 
data to be gathered during the procedure, how to arrange the data so it becomes 
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useful information to the Captain, and what to say when actually making the 
report. Non-behavioral objectives enhance focus on both specific and non¬ 
specific aspects of a lesson. I believe using behavioral objectives in a training 
environment is the better organizational policy. In doing so, the students remain 
focused on the most important aspects of their training. Horton (2000) advocates 
writing the objectives in terms of what the student will do, say, or write as 
opposed to providing the learning objectives as teaching objectives (from an 
instructors point of view), which stresses the application of training in a 
meaningful way to the students (p. 91). Horton (2000) identifies phrases and 
words that are good to use, and those that are undesirable when writing learning 
objectives. When generating learning objectives, Horton (2000) recommends 
using words such as create, make, install, set-up, edit, revise, amend, start, 
repair, replace, diagnose, troubleshoot, organize, and write (p 91). Words that 
should be avoided include understand, list, give an example of, articulate, 
explain, describe, identify, recall, master, and differentiate (p. 91). According to 
Horton (2000), these words are to be avoided because they describe student 
performance in terms of what the instructor is teaching, not in terms of what the 
student must do to meet the objective. 

c. Learning Materials / Instructional Information 

Learning materials include books, manuals, videotapes, DVD’s, 
CD-ROM’s, and all other media that assists the student in learning. Each of the 
aforementioned media all have one thing in common. They each present 
instructional information to the student. How they do this is also important. 

According to Allessi and Trollip (1991), there are four types of 
information that can be effectively presented to students via CBT. They are (a) 
verbal information, (b) general concepts, (c) rules and principles, and (d) skills (p. 
44). I have reservations about the utility of teaching skills by way of CBT. Of 
these four types, a set of skills is the most difficult to teach via the CBT method. 
In order to effectively teach a new skill set, it may be necessary to teach the 
concepts, rules, and principles that guided the development of the skill. CBT is 
an appropriate and proven vehicle to present the first three types of information, 
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but it may not be the best mechanism to teach skills, especially if CBT is used in 
conjunction with on-the-job-training. From experience, it is easier to learn a new 
procedure, especially when it involves operating machinery, when the trainee is 
able to see, touch, hear, smell, and manipulate the actual equipment. As a 
general rule, it is best to teach from the simple to the complex. This means 
ideas, concepts, then rules and principles, followed by the actual procedure 
should be the sequence in which skills are taught. 

d. Feedback Loop 

The feedback loop is a very powerful learning tool educators have 
at their disposal. According to Stolovitch and Keeps (2003), Allessi and Trollip 
(2001), and Wiggins (1998), feedback to students should depend on their 
answers and needs to be either corrective or confirming. Wiggins (1998) states 
corrective feedback will compare current student performance to what a 
successful outcome is, which focuses student attention on how to improve (p. 
49). Stolovitch and Keeps (2003) argue confirming feedback is used as a reward 
for answering questions correctly, and that frequent and specific feedback has 
been demonstrated to improve performance (p. 67). 

5. Instructional Presentation 

There are three basic aspects to a CBT delivered presentation I will 
examine in this section. They are the screen organization and use of graphics; 
text, language, and grammar; and mid-topic questioning techniques. Screen 
organization centers around the layout of the screen during a lesson. 
Additionally, basic guidelines for the use of graphics will be presented. The 
proper use of language and grammar are essential to the development of a high- 
quality training program. The way in which textual information is included in a 
presentation can affect learning. Simple guidelines for presenting text will be 
explained. An important aspect to any training program is the use of questions 
throughout the lesson to determine student comprehension and the associated 
feedback to the students about their performance. Successful questioning 
techniques will also be examined. 
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a. Screen Organization 

According to Berry (2000), research suggests that visuals (web 
pages or CBT pages) are equivalent to lengthy text in their ability to effectively 
communicate (p. 48). Leflore (2000) details three laws that are important to know 
if developing CBT software. By understanding the principles encompassed by 
these laws, developers can begin to place instructional objects into courseware 
in ways that are very beneficial to the students they are training. The first is the 
Law of Proximity, which states it is easier for learners to understand that different 
text or graphic elements go together if these elements are placed close together 
(p. 103). For example, the text that describes a figure or illustration should be 
placed close to that illustration. The second law Leflore (2000) describes is the 
Law of Similarity (p. 103). This law states that people will group things together 
that look similar (p. 103). For example, if elements in a graphic are all of the 
same style, the graphic will be seen, or perceived, as a whole (p. 103). The third 
law Leflore (2000) mentions is the law of Closure. Leflore (2000) states 
individuals will try to interpret incomplete graphics or text based upon past 
experience (p. 104). 

Leflore (2000) also provides a set of six Gestalt Theory guidelines 
appropriate for following when constructing a CBT delivered course. They are: 

(a) When designing web-based instruction, make sure that the background does 
not interfere with the clarity of the information presented in the foreground (p. 
104). 

(b) Use simplified graphics to introduce new material. If the concept requires 
complexity, start simple and gradually add in the complexity (p. 104). 

(c) Place related information together on the screen so the learner will 
automatically grasp the information together as a unit rather than as separate 
elements (p. 104). 

(d) Use color, animation, flashing, or other means to draw attention to key 
phrases in text or areas in graphics (p. 104). 
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(e) Ensure text and graphics are complete, so the student does not have to 
spend energy “making” or “creating” meaning (p. 104). 

(f) When introducing a new topic, vocabulary unique to the content should be 
introduced by using common terms. Avoid using jargon until after all the new 
terms have been introduced (p. 104). As a matter of style, it is important that 
courseware developers avoid using acronyms without first explaining them 
(Allessi and Trollip, 1991, p. 43). 

It is disconcerting to students when they must “scroll” through 
lengthy text. According to Allessi and Trollip (1991), it is hard for people to 
distinguish between old and new information when it is moving (p. 35). 
Therefore, it is important to limit the amount of textual information that is 
presented to no more than one page at a time to avoid making the students scroll 
through text. Allessi and Trollip (1991) also recommend enclosing the primary 
text in a box, for emphasis, when using both graphics and text on the same page. 
The students’ attention will then become focused on the most important 
information (p. 38). 

b. Graphics 

Graphics are very useful tools that can be used for the presentation 
of instructional information, presentation of analogies, or as cue’s for student 
action. Belanger and Jordon (2000) detail two types of graphical animation: 
bitmapped or object-oriented. A bitmapped image is drawn, and motion is 
generated through showing progressive movement in frames per second (EPS). 
Full motion video is shown at 30 FPS and a minimum of 10 FPS is required to 
avoid a “choppy”, or strobe effect (Belanger and Jordon, 2000, p. 130). 

Complex graphics can easily confuse students unfamiliar with the 
material. Graphics that contain a lot of information should be, if possible, broken 
down into smaller parts and presented sequentially (Allessi and Trollip, 1991, p. 
40). Allessi and Trollip (1991) also contend that graphics should always be 
presented with their associated text because changing back and forth between 
an illustration and its textual description makes comprehension more difficult for 
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the student (p. 40). According to Williams, Paprock, and Covington (1999), 
designers can also help the students by using graphical displays that are “off- 
centered”. This forces the students to scan the whole screen and not just focus 
on the center of it (141). 

1. Color. The use of color can add breadth to any 
presentation, increasing student interest and motivation. For example, using 
different colors to represent segments of a pie chart can be an effective way to 
help students quickly distinguish between each segment of the graph. Using a 
colored screen may indicate to students that an action must be taken. If the 
computer is awaiting student input, the background color of the screen could be 
yellow. The screen could flash red or green depending on whether a student 
answered a question correctly. Although the use of color is encouraged, there 
are a few rules to be followed when using color in a CBT environment. 

Allessi and Trollip (1991), and Williams, Paprock, and 
Covington (1999) agree that if multiple colors are used on the same screen, they 
should contrast. Additionally, red and blue should be used sparingly since they 
are not perceived as well as colors in the middle of the visible spectrum, such as 
yellow and green. Dix et al. (1997) add that the colors used should correspond 
to common conventions and user expectations (p. 115). This means red should 
mean “wrong”, or “stop”, whereas green should mean “correct”, or “go”. Dix et al. 
(1997) also argue if color is used as an indicator, it should not be the only cue 
given to the student. For example, if a course used a colored screen to cue a 
student that a question is answered correctly or not, it should also include the 
words “Correct”, or “Wrong” within the colored screen as a second cue to the 
student. 

c. Text, Language, and Grammar 

There are a number of factors that must be considered when 
presenting textual information via a computer. These include the characteristics 
of the text itself, textual quality, along with language and grammar. 

Williams, Paprock, and Covington (1999) offer a set of guidelines 

appropriate for the way text should “look”, or its characteristics, when using CBT 
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to present information. Williams, Paprock, and Covington (1999) state that since 
the art of turning verbal information into visual communications involves working 
with space, text, fonts, and visual relationships, it is best to use a simple bold 
“Sans Serif” fonts (p. 139). The bold type makes the letters sharper. Allessi and 
Trollip (1991) and Williams, Paprock, and Covington (1999) believe it is best to 
use a mixture of capital and lowercase letters, and avoid using all capital letters. 

For best readability, Williams, Paprock, and Covington (1999) 
recommend using a maximum of seven lines in height and seven words in length 
per line on a given presentation page (p. 139). Using these guidelines forces 
educators to be brief and concise, a trait often expected of their students. 
According to Allessi and Trollip (1991), all CBT presentations should say just 
enough to teach what is desired (p. 43). 

The quality of text is an important concept. Not only must the font 
be readable and the text brief, but it must precisely convey the main ideas of the 
lesson so that the learning objectives will be met. Pronouns with unclear 
referents should not be used. Additionally, more than one word should not be 
used to refer to the same thing and the use of the same word at different times to 
mean different things should be avoided (Allessi and Trollip, 1991, p. 43). 

It should go without saying that CBT presentations should be inherently 
flawless with respect to spelling, grammar, and punctuation. This is not always 
an easy task considering that brevity is also considered a necessity. The key to 
enhancing student learning and comprehension is consistency throughout the 
course of instruction. 

d. Mid-Topic Questioning Techniques 

According to Allessi and Trollip (1991) and Gronlund (1998), 
questions asked during a lesson serve many functions. Mid-topic questions help 
keep the student attentive, provide needed practice, encourage deeper 
processing, assess student recall and understanding, and provide a basis for 
lesson sequencing. Each of these is an appropriate reason to include 
questioning within the framework of each topic. I recommend asking questions 
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about material about to be taught as well as material that has already been 
taught. Asking questions about material yet untaught will focus the student’s 
attention on the material he or she is about to learn. Questioning students about 
material already presented will assist in verifying the learners’ comprehension. 

It can be difficult to know how much questioning should be used 
during a lesson. Allessi and Trollip (1991) argue that it should occur frequently 
because the more a student interacts with the program, the more attention will be 
maintained and more learning will be facilitated. Although I agree frequent 
questioning will improve attention, I believe too much questioning can lead to 
student frustration. Too many questions during a lesson may be interpreted as 
unnecessary interruptions that distract and anger students. I recommend using 
caution when placing questions in the middle of a presentation. Having been an 
enlisted naval instructor for nearly four years, I have come to believe it is better to 
ask a few questions during the lesson to maintain the students’ attention and 
then ask deeper thought provoking questions at the end of each topic. This 
technique worked very well for me in the classroom, and I believe it would work 
just as well in a CBT environment. 

6. Student Testing 

Assessing student performance is a vital aspect of all training programs. 
Assessments can be summative or formative. Summative assessments are 
completed at the end of a course whereas formative assessments are used to 
monitor progress throughout the course. 

a. Testing Schemas 

There are three basic testing schemas. In the first, the computer 
grades students’ answers. Horton (2000) argues this technique has two 
advantages. Evaluation is immediate and the computer is nonjudgmental (p. 
277). However, Horton (2000) and Allessi and Trollip (1991) report that because 
the computer is limited in the types of answers it can grade, teachers can only 
ask simple forms of questions. The second schema utilizes the student’s 
coworkers or managers to grade exams. According to Horton (2000), the 
disadvantage to this approach is coworkers may not be available or may not 
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have the expertise required to evaluate the learner (p. 277). The last schema 
calls for the student’s answers to be transmitted to the instructor for grading. 
Because instructors can spot any subtle mistakes and grade appropriately, there 
is no limit to the types of questions that can be asked of students (Horton, 2000, 
p. 277). 

b. Test Construction 

Ciavarelli (2003) outlines four rules to follow when building tests. 

They are: 

(1) Examination content must be based upon the learning objectives (p. 14). 

(2) Performance requirements for the examination must be consistent with the 
learning objectives (p. 14) 

(3) Examination instructions must be clear (p. 14). 

(4) Feedback regarding the examination must be timely, constructive, and 
relevant (p. 14). 

c. Remediation 

Remediation is the follow-on presentation of the same instructional 
material to a student who did not meet the necessary standards on an 
examination or performance review. Allessi and Trollip (1991) state that 
remediation can occur in the form of repeating information already seen or the 
form of providing information in more detail (p. 77). This can include giving the 
student more examples, pictures, sample problems, or practice problems. 

C. EVALUATIONS 

Evaluation is essential to any educational endeavor. There are two 
fundamental types of evaluations: course (or program) evaluation and student 
evaluation. Both are necessary, since one type of evaluation examines whether 
the right material was taught to the students in an effective and efficient way, and 
the other examines whether the students learned the material sufficiently well 
enough to conclude the learning objectives were met. As can be seen, 
evaluations determine the effectiveness of any training program. 
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Authors that write about evaluations often quote Kirkpatrick. Many 
consider him a pioneer in the field. Kirkpatrick (1998) argues, and Dix et al. 
(1997) concur that there are three primary reasons to evaluate training. First, to 
justify the existence of the training department by showing how much it 
contributes to an organization’s objectives and goals. Second, evaluations are 
used to decide whether to continue or discontinue a training program. Third, 
evaluations are used to gain information on how to improve future training 
programs. 

Whatever type of evaluation is being constructed, it needs to be 
contextually relevant (Moskal and Dziuban, 2001, p. 161). There are guidelines 
to be followed to ensure evaluations generate valuable and useful information to 
be used to make the training program better. The expense of time and money to 
perform an evaluation is wasted if the correct information is not being collected. 

1. Training Objectives 

Nearly half a century ago. Bloom (1956) identified three educational 
domains. They are the cognitive domain, the affective domain, and the 
psychomotor domain (p. 7). Learning objectives within the cognitive domain 
focus on recall of knowledge and the development of intellectual skill (p. 7). 
Learning objectives within the affective domain focus on changes in interest, 
attitudes, and values (p. 7). Learning objectives in the psychomotor domain 
focus on manipulative, or human motor-skills (p. 7). 

Traditionally, student performance within each of these domains is 
evaluated differently. Performance in the cognitive domain is usually measured 
through tests and papers. In a traditional college classroom this equates to a 
midterm, a final exam, and a moderately sized research or opinion paper. It is 
harder to evaluate students across the affective domain because it is harder to 
objectively evaluate a student’s attitudes. Teachers attempt to achieve this by 
taking attendance and giving a grade for in-class participation. Performance 
based tests, or demonstrations, are normally used to evaluate students’ 
psychomotor skills. 
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a. Bloom’s Taxonomy 

Bloom (1956) developed a taxonomy, or method of classifying 
educational objectives. Bloom (1956) defined an educational objective as the 
explicit formulations of the ways in which students are expected to be changed 
by the educational process (p. 26). Each category that is presented, builds on 
the previous one. A student can be thought of as moving through these 
categories, from one to another, as knowledge and intellectual skill improve. 
Additionally, the way questions are worded when assessing student achievement 
varies from one objective category to the next. Bloom’s (1956) taxonomy of 
objectives includes the following categories: knowledge, comprehension, 
application, analysis, synthesis, and evaluation. 

Knowledge is remembering, by recall or reorganization, some idea 
or phenomenon with which the student has had experience in the educational 
process and it is basic to all other ends or purposes of education (Bloom, 1956, 
p. 18). Basic questions that test recall are often used to assess knowledge. For 

example, “A destroyer is a type of_?” would be an appropriate question if 

testing in the knowledge domain. 

Comprehension is related to behaviors or responses that represent 
an understanding of the literal message contained in a communication (Bloom, 
1956, p. 89). Asking students to translate or interpret ideas are effective means 
to assess student comprehension. A question asking students to illustrate the 
concepts depicted on a chart or other graphic would be an effective way to test 
comprehension. 

Bloom (1956) defines application as the illustration of a task or idea 
that requires the comprehension of a method, theory, principle, or applied 
abstraction (p. 120). Asking a student to explain in detail the underlying reasons 
why a modern ship ran aground after providing the student with a set of facts 
about the case in question is an example of how to assess student capabilities 
within this domain. 
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The category of analysis is similar to application. Analysis is the 
breaking down of the given material into its constituent parts and detection of the 
relationships of the parts and the way they are organized (Bloom, 1956, p. 144). 
Giving students an article to read and then asking questions about the article, 
like, “Which of the following best describes...” or “The argument that was 
presented was flawed because...” is effective at measuring students’ analytical 
proficiency. 

Bloom (1956) describes synthesis as the putting together of 
elements and parts so as to make a whole (p. 162). He argues that it is a 
combining parts and elements in such a way as to constitute a pattern or 
structure not clearly there before (p. 162). Conducting research and then 
developing a template for analyzing CBT could be considered an exercise in 
synthesis. Requiring students to set a poem to music, write a simple melodic 
line, or write a composition with a single tonal base are all examples of testing 
students within the synthesis domain (Bloom, 1956, p. 179). 

Bloom’s last objective is evaluation. Bloom (1956) defines 
evaluation as making judgments about the value, for some purpose, of ideas, 
works, solutions, methods, or materials (p. 185). According to Bloom (1956), 
evaluation involves the use of quantitative and or qualitative criteria as well as 
standards for appraising the extent to which particulars are accurate, effective, 
economical, or satisfying (p. 185). 

These domains are as important today as they were fifty years ago. 
With respect to training, organizations know to what extent they want an 
employee to be trained in a specific job. Some employees only need have a 
simple understanding, or comprehension, of their job tasks. A gas station 
attendant, for example, may only need to know how to pump gas, check the oil, 
and clean automobile windows. Other employees need to be able to assimilate 
obscure and seemingly unrelated information and make a decision that may 
impact the organization for years to come. It is likely that these employees will 
receive training that will enhance their ability to function in the higher (synthesis 


49 



and evaluation) domains. It is important that organizations build mechanisms 
into their courses of instruction that verify (1) that the course’s learning, or 
educational, objectives are in agreement with organizational goals, and (2) that 
the students’ performance after training exceeds the standards set by the 
educational objectives. 

This is not an easily accomplishable task. Kirkpatrick (1998) 
contends that evaluating the results of training is very difficult (p. 60). “What is 
the amount that quality or productivity improved as a result of training?” or, “What 
is the amount training contributed to profits?” are questions that many 
organizations have historically been unable to answer (Kirkpatrick, yr, p. 60). 

However, this does not mean that it is impossible to validate 
training objectives. One method, although time consuming and rigorous, that has 
been proven successful by Housel and Bell (2001) in answering many 
Kirkpatrick’s questions. It is the Knowledge Value Added (KVA) method. 
Organizations that properly utilize KVA are able to determine if their training, and 
specifically, what training is contributing to improved or decreased revenues. 
Although very important, it should be noted that this is just one dimension of 
course or curriculum evaluation. 

2. Evaluation Design 

According to Bloom (1956) and Renshaw and Taylor (2000), the design 
and evaluation of computer-based instructional materials should be 
accomplished with appropriate memory models and learning theories as a 
reference. Despite the importance of and vital need to conducting evaluations, 
there is no universally agreed upon standards or methods for their performance. 
Some, like Wiggins (1998) seem to advocate more of an emphasis on student 
evaluations, while others like Kirkpatrick (1998) seem to focus on program 
evaluations. Despite the differences in their approaches, the goal remains the 
same: ensuring that the learning objectives are met. 

Hall (1998) suggests that three themes should guide the development of 
any evaluation model. First, Hall (1998) contends that an overriding direction. 
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taking into account learners, contexts, and goals, should be meticulously 
identified as a first step (p. 138). Next, Hall (1998) argues that proper design is a 
matter of finding the proper balance between elements of simplicity and 
complexity (p. 138). In other words, do not overdo it. Last, Hall (1998) states 
that evaluations and assessments must be part of any program design process 
because without evaluations, the designer will never become aware of how 
effective a training program is (p. 138). 

Wiggins (1998) offers six criteria that should be examined during an 
evaluation. According to Wiggins (1998), all assessments should be: 

(a) Credible to all stakeholders (p. 111). 

(b) Useful, helpful to teachers, students, and administrators (p. 111). 

(c) Balanced in the use of assessment methods, to provide a rich, 
defensible, and feasible profile of achievement (p. 111). 

(d) Honest, yet fair (p. 111). 

(e) Intellectually rigorous and thought provoking; focused on core ideas, 
questions, problems, and knowledge (p. 111). 

(f) Feasible in terms of resources, logistics, politics, and redeployment of 
time for collaboratively designing, debugging, using, evaluating, and effectively 
reporting student work (p. 111). 

Compare these student-focused criteria against a set of program-centered 
criteria. Kirkpatrick (1998) contends that there are eight broad factors that should 
be considered when performing an evaluation of a program. They are: 

(a) To what extent does the subject content meet the needs of those 
attending (p. 17)? 

(b) Is the instructor the one best q ualified to teach (p. 17)7 

(c) Does the leader use the most effective methods for maintaining 
interest and teaching the desired attitudes, knowledge, and skills (p. 17)7 

(d) Are the facilities satisfactory (p. 17)? 
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(e) Is the schedule appropriate for the participants (p. 17)? 

(f) Are the aids effective in improving communication and maintaining 
interest (p. 17)? 

(g) Was the coordination of the program satisfactory (p. 17)7 

(h) What else can be done to improve the program (p. 17)7 

It is obvious that each of the previous two lists were designed to evaluate 
different aspects of a course of instruction. The objectives in each case, 
however, were the same, to ensure that the students learned what the instructors 
intended them to. To argue for the primary use of one of these two approaches 
over the other is counterproductive. Both types of evaluations are essential to 
ensuring not only that the students are learning, but that they are learning the 
right material. I agree with Belanger and Jordon’s (2000) argument that the key 
consideration for the success of any distance-learning environment is whether or 
not, and to what degree, the learning objectives are being met (p. 14). 

3. Pedagogical Evaluation 

There are many evaluation methods, or models, available for use. Some 
organizations hire other firms to conduct training evaluations, while others 
develop a set of in-house guidelines. Some approaches to evaluation are 
simple, and some are very complex. Some companies even prefer to use a 
multifaceted approach. What is clear is that the approaches, methods, 
evaluators, and scoring rubrics vary greatly. 

Belanger and Jordon (2000) report that determinants of program success 
can be grouped into the four categories of institutional characteristics, learner 
characteristics, course characteristics, and distance learning characteristics (p. 
187). Institutional characteristics factors include institutional objectives, delivery 
mechanisms, and support structures (Belanger and Jordon, 2000, p. 187). 

Learner characteristics include the factors of motivation, desire to excel, 
and self-sufficiency (Belanger and Jordon, 2000, p. 188). The course 
characteristic factors include group project, final paper, knowledge based or 
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performance-based criterion (Belanger and Jordon, 2000, p. 188). Distance 
learning factors consist of the type of technology and learning environment. 

This structure provides a decent framework for building a comprehensive 
evaluation template from. This model would be very useful in making baseline 
predictions about the future success or failure of a training program. Upon 
examination, this model does not define any of the factors that fall into the four 
categories. Just as it is important that organizations adequately define their 
educational objectives, it is also important that organizations precisely define the 
measurement categories, factors, and rubrics that will be used to evaluate the 
training program. 

Parikh and Sameer (2002) describe an empirical model for program 
evaluation that is similar to Belanger and Jordon’s. This model also has four 
dimensions. Parikh and Sameer (2002) list them as content, technology, 
interface, and functionality (p. 35). The content section of this model measures 
the characteristics of the information provided by the system to the users, how 
precise the information is, what is its value, and if it meets the users’ needs 
(Parikh and Sameer, 2002, p. 35). The technology category is used to 
determine whether the organization is utilizing appropriate technologies to deliver 
the right information at the right time (Parikh and Sameer, 2002, p. 35). Interface 
examines whether the system is easy to learn and user friendly (Parikh and 
Sameer, 2002, p. 35). Parikh and Sameer (2002) state that evaluators would 
use this section to verify that the screen layout is appropriate, instructions clear, 
and information is presented in a useful format (p. 35). Functionality measures 
the practicality as opposed to the attractiveness of the system (Parikh and 
Sameer, 2002, p. 35). 

4. Instructional Evaluation 

All instructional activities need to be assessed. It is possible students are 
being taught old, or outdated materials. Unfortunately, it is possible that an 
instructor, even though an expert, may not be very good at teaching. Tests can 
be biased toward a particular race or gender. If organizations did not evaluate 
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the various aspects to instruction, these potential problem areas might never be 
discovered. If never discovered, they cannot be corrected. 

Gronlund (1998) believes assessments allow organizations to evaluate the 
effectiveness of three parts of the instructional process. Organizations can 
ascertain to what extent instructional objectives were realistic, evaluate whether 
methods and materials of instruction were appropriate, and determine how well 
the learning experiences were sequenced (p. 11). Gronlund (1998) offers a set 
of guidelines to follow when performing instructional assessments and analyzing 
the results: 

(1) Organizations must have a clear conception of all intended learning 
outcomes (p. 18). What types of knowledge, understanding and application does 
the organization want students to gain and what evidence is the organization 
willing to accept as proof that the students have achieved to that level (p. 18). 

(2) The organization should use a variety of assessment procedures to 
verify student achievement (p. 18). These can include tests, quizzes, surveys, 
performance assessments of teachers and students, or a combination of these 
methods (p. 18). 

(3) The instructional relevance of the procedures must be considered (p. 
18). Consider what happens if a student completes a CBT lesson, but the 
instructional material only covered three of the five learning objectives. Gronlund 
(1998) argues, and I agree, the instruction has lost relevance because the 
learning objectives, instruction, learning tasks, and assessment procedures are 
not in close agreement. There is hardly a greater “instructional sin” than telling 
students that they will be required to learn material to a organizationally defined 
standard and then not allowing them to meet that standard because not enough 
information, or the wrong information, was presented. 

(4) Specifications of criteria for judging successful performance must be 
known and published (p. 20). Without clearly defined and published grading 
criteria, it is easy to demotivate students. For example, assume a class of 
students is required to complete an essay, but they have not been told what the 
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grading criterion is going to be. When the papers are handed back, the students 
that did not receive the highest grades will not only wonder if the papers were 
graded fairly, but will also lose motivation and interest because they are unsure 
what how to write an “A” paper next time. 

(5) Feedback must emphasize the strengths of performance and any 
weaknesses that need to be corrected (p. 21). 

(6) A comprehensive grading and reporting system must be in place (p. 
21). Gronlund (1998) infers that letter grades should be eliminated (p. 21). 
Letter grades help educational institutions separate good performers from poor 
ones, and have value in public education. However, in an organizational training 
environment, letter grades mean very little. Employees would be better served if 
they were provided specific, objective, relevant, and non-biased feedback that 
praised their strengths, identified their weaknesses, and concluded by offering 
suggestions for improving performance. Furthermore, it this feedback would be 
very valuable if it were given many times throughout a lengthy course of 
instruction, as opposed to once at the end of the class. 

D. COMPREHENSIVE MODEL FOR THE EVALUATION OF CBT 

This CBT evaluation framework consists of two sections. This first section 
is Pedagogical Evaluation. The second section is Instructional Evaluation. Each 
section consists of four functions. The Pedagogical Evaluation section contains 
a Program Operation function, an Instructional Presentation function, an 
Assessment Function and a Feedback function. The Instructional Evaluation 
section consists of a Course and Learning Objectives function, a Core 
Curriculum function, along with Assessment and Feedback functions. Figure 4 
displays a representation of this evaluative framework. Two sets of 
questionnaires were developed for each section. One set is designed for 
developers and program evaluators to use, the other set is designed to gather 
information from students. Appendix A contains all of the questionnaires. 
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Two elements, Interactivity and Instructional Sequence, overlap the 
boundaries of the two main sections. These elements fit best within the 
Pedagogical Evaluation section’s Instructional Presentation function, but cannot 
be properly evaluated without taking into account the Educational Objectives 
function’s learning objectives and the instructional content of the Core 
Curriculum. 

The Instructional Quality element assesses the overall quality and 
effectiveness of the CBT initiative. The strengths and weaknesses of the 
Pedagogical Evaluation and Instructional Evaluation sections are compared and 
the instructional effectiveness of the CBT initiative is determined. In other words, 
it is a gage of how well the CBT courseware meets organizational training 
objectives through the proper presentation, monitoring, and assessment of 
instructional content, and how well feedback is provided to both the organization 
and its students. 

1. Pedagogical Evaluation. 

This section of the evaluation framework will assess the various aspects of 
CBT that are not associated with the information that is taught and the way 
information is tested throughout the student population. The information 
gathered using the functions in this section will enable organizational leadership 
and those who develop CBT programs to effectively identify the strengths and 
weaknesses inherent in the techniques their CBT software programs’ are 
installed, used, and presented to students. Once deficiencies are identified, 
organizations can, based upon their needs or budgetary constraints, implement 
procedures that will eliminate or minimize the identified weaknesses, 
a. Program Operation 

The Program Operation function examines the software’s interface 
and amount of control that each learner has while using the tutorial. 

1. Interface. The interface between the student and the 

computer is very important. It determines what the student is going to look at on 

the computer screen while navigating through each lesson. This element 
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critically examines screen organization, student control during training, ease of 
installation and ease of use. 

2. Security. This element assesses whether students’ 
personal information, test scores, or performance evaluations are protected from 
intrusion, tampering, or theft. 

b. Instructional Presentation 

The Instructional presentation function assesses the quality of CBT 
instruction. This function will not evaluate the instructional content of a lesson; 
only how well it is presented. 

1. Interactivity. This element evaluates whether a CBT 
lesson is interactive. Although the objective of this element is not to evaluate 
instructional content, whether a lesson contains an appropriate amount of 
interactivity is determined in a large part by the instructional content’s length and 
complexity. 

2. Lesson Sequencing. The lesson-sequencing portion of 
this section assesses whether or not lesson topics are properly sequenced. 
Although the objective of this element is not to evaluate instructional content, 
whether a series of modules or lessons are correctly sequenced is largely 
determined by lesson content and the principle of teaching from simple to 
complex. 

3. Multimedia Format. This element evaluates whether 
incorporated graphics, animations, cartoons, video, and audio, is being used 
correctly. 

4. Color. This element evaluates whether the use of color 
has been implemented correctly. 

5. Text. This element verifies that text has been properly 
laid out and spaced, and that the presentation is grammatically correct at an 
appropriate reading level. 

6. Questioning. This element assesses whether appropriate 
questioning techniques have been implemented throughout the lesson. 
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7. Environment. This element analyzes the suitability of the 
students’ learning environment while using CBT. 

c. Assessment 

This function of the pedagogical evaluation section evaluates how 
the organization conducts assessments. 

1. Methods. This element evaluates whether organizational 
assessment methods are defined and used. 

2. Rubrics. In this instance, a rubric is the standard with 
which each function of the CBT program will be evaluated. This element 
evaluates whether rubrics for each “area” the organization will assess have been 
properly defined. 

3. Participants. This element determines if all of the 
evaluation participants, along with each one’s role in the evaluation has been 
defined and promulgated. 

4. Usefulness. This element assists in evaluating whether or 
not the information collected during an assessment is useful to the organization. 

d. Feedback 

Feedback regarding the pedagogical effectiveness of any 
organizational training initiative is critical for the success and continued 
improvement of that venture. This function evaluates whether the feedback to 
the organization regarding their CBT’s pedagogical effectiveness is timely, 
relevant, and useful. 

2. Curriculum Evaluation 

The Curriculum Evaluation section of this model will analyze the course 
content for relevancy, correctness, and usefulness to both the organization and 
those undergoing the training. 

a. Learning Objectives 

This element of the model will verify several things. First, that the 
organizational goals with respect to a particular training program are defined. 
Second, it will determine if the training program’s learning objectives have been 
derived from the program’s stated goals and if the students’ performance 
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standards were derived from the learning objectives. Third, it will verify that the 
learning objectives are presented in the CBT program in terms of student 
performance. 

b. Core Curriculum 

This function will verify that the instruction that the students receive 
is relevant, current, and useful to the students. It will also verify that as a result 
of this instruction, students are able to meet or exceed all required performance 
standards. 

c. Assessment 

This function of the pedagogical evaluation section evaluates how 
the organization conducts assessments. 

1. Methods. This segment evaluates whether assessment 
methods for each lesson are defined and used. 

2. Rubrics. This segment evaluates whether rubrics for each 

test or performance assessment have been properly defined. 

3. Participants. This segment determines if all of the 
participants, along with each one’s role in the test, quiz, or performance 
assessment has been defined and promulgated. 

4. Usefulness. This segment assists in determining whether 
or not the information collected tests and performance evaluations are useful to 
the organization. 

d. Feedback 

Feedback regarding the instructional effectiveness of any 
organizational training initiative is essential for the success and continued 
improvement of that venture. This function evaluates whether the feedback to 
the organization and the students regarding the CBT instruction and testing is 
timely, relevant, and useful. 

3. Conclusion 

Use of this template and its associated survey sheets found in Appendix A 
will allow evaluators to conduct comprehensive assessments of their CBT 
programs. The survey sheets will be used to conduct an evaluation of the 
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reengineered CBT Surface Warfare Officer Division Officer course of instruction. 
The result will be an accurate assessment of the course’s cognitive and 
pedagogical strengths and weaknesses. 

With minor modifications to the pedagogical evaluation section of 
template, it could be effectively used to evaluate the strengths and weaknesses 
of traditional classroom instruction. Specifically, computers differ from humans in 
the way in which they present information. Since they present instructional 
materials differently, different criteria for conducting that portion of an evaluation 
is required. 
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IV. ANALYSIS 


This analysis is being conducted to fulfill two objectives. First, it acts as a 
proof of concept that the evaluation framework presented in the previous chapter 
works. Second, it depicts an accurate snapshot of the SWOS Division Officer at 
Sea Training Initiative at the present time. Since its inception in January 2003, 
very few, if any junior officers have completed this training initiative and earned 
their Surface Warfare Pin. This is to be expected since the majority of newly 
accessioned officers just joined the fleet in June and July of 2003. 

A. DATA COLLECTION METHOD 

1. Participants 

Officers stationed aboard ships homeported in San Diego, California 
completed the surveys in August 2003. 17 Ensigns completed the student 

questionnaire and four Training Officers completed the supervisor survey. These 
officers were promised complete anonymity in exchange for their honesty in filling 
out the surveys. Consequently, ship types, names, and hull numbers are omitted 
from this report. Small group discussions were held one each ship immediately 
after the surveys were completed. 

2. Survey Instruments 

Two surveys were generated to assist an evaluator assessing a CBT 
program using the framework developed in chapter III. One survey is tailored for 
course administrators and one for students. The survey questions were 
generated through multiple means. Ouestions were developed based upon the 
literature review and the principles discussed in Chapters II and III. Dr. Anthony 
Ciavarelli, an expert in the field of organizational training who teaches at the 
Naval Postgraduate School, Monterey, also contributed questions. These 
questions started with a baseline survey assessing the quality of online 
instruction derived from Ciavarelli (2003). Several questions are worded in such 
a way as to only be applicable to students learning in a shipboard environment, 
but with minor modifications, could easily be asked to students in any CBT 
learning environment. Each of the two surveys had three sections. Section one 
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asks basic background information. Section two presents multiple statements 
and asks students to indicate whether they agree or disagree with the statement. 
A Leikert scale from one to five is used in this section. Section three asks 
students to provide short essay answers to multiple questions. A complete 
analysis of each segment in the Evaluation Framework will not be provided. 
Instead, areas that have identified strengths or weaknesses will be discussed in 
depth. Complete survey data is provided in Appendix B. 

B. TRADITIONAL CLASSROOM SWO TRAINING 

The curriculum that was used in the classroom to train Ensigns designated 
to become Surface Warfare officers was converted, in its entirety, to a computer- 
based training program that is now loaded onto computers from compact discs. 
In order to effectively determine whether the new teaching method is as effective 
as the one it replaced, an overview of the strengths and weaknesses of the 
historical classroom training program is necessary. 

Historically, the Surface Warfare Officers School, Division Officer 
Curriculum (SWOS DOC) made extensive use of the classroom environment to 
train newly commissioned Ensigns who were to be permanently assigned to the 
Navy’s Surface Warfare community. The Surface Warfare classroom curriculum 
structure has been modified several times over the years, but the information 
taught to those who received the training remained, for the most part, constant. 

Instruction typically began at 0800 and usually concluded near 1600. 
Each fifty minutes of instruction ended with a ten-minute break. Students could 
expect to receive a daily lunch break that would last between thirty minutes and 
one hour, usually between 1100 and 1200. The students’ daily routines were 
typical for most naval personnel, both officer and enlisted, who have received 
formal classroom training during the past twenty years. 

1. Core Phases of Instruction 

The Surface Warfare Officer School’s website states the Surface Warfare 
curriculum was designed to prepare newly commissioned Ensigns, enroute their 
first tours as division officers afloat, to stand in port and underway watches, and 

to manage the administrative duties of the division officer afloat. The curriculum 
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was divided into three phases: Core Phase I, Core Phase II, and Billet Specialty 
Training. Core Phase I consisted of eleven weeks of operations and combat 
systems training, Core Phase II consisted of six weeks of platform specific 
engineering training, and Billet Specialty Training was comprised of three to six 
weeks of tailored operations, combat systems and engineering training specific to 
the trainee’s prospective shipboard billet. 

The Core Phase I curriculum was divided into units. These units were 
subdivided into modules, which were then broken down further into topics. It is 
the topics that were taught in the classroom. Including tests, quizzes, and 
reviews; the Core Phase I curriculum had 186 topics that were taught. Appendix 
C contains a listing of these topics taken from Intelligent Design System’s, Inc.’s 
(IDSI) Final Report. 

Core Phase II was divided into six units. Course content varied; 
depending on the class of ship the student will be going to. For example, a 
student with orders to an Oliver Flazard Perry class frigate (FFG) would receive 
instruction that covers the basics of a gas turbine propulsion plant and a student 
with orders to an Austin class Landing Platform Dock (LPD) would receive 
instruction in basic steam propulsion. 

Billet Specialty Training was comprised of one or more service schools 
that provided the junior officer with the basic information that he or she must 
know in order to satisfactorily perform their assigned jobs once they arrive 
aboard their first ship. For example, a prospective frigate Combat Information 
Center Officer (CICO) is usually also the ship’s Electronic Warfare Officer 
(EWO). This officer would first receive a week’s training in CICO School, 
followed by a week of training in EWO School. At the completion of this training 
it is assumed that the Ensign possesses enough basic knowledge about the 
duties and responsibilities of the job that he or she will be able to immediately 
perform adequately in that role with minimal supervision. 

It is important to remember that although it is difficult to remember all of 
the information presented in a six-month course of study, the information that this 
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curriculum imparted is important because it sets the knowledge foundation on 
which the new officer will build over the course of a military career. Once newly 
commissioned officers reported aboard their first ships, further training drew out 
and reinforced what was learned at SWOS. Additionally, SWOS presented 
newly commissioned officers with all of the information required by the Personnel 
Qualification Standards (PQS) to successfully qualify to stand the two watch 
stations that are needed to earn the Surface Warfare Pin: Combat Information 
Center Watch Officer and Officer of the Deck Underway. SWOS, however, did 
not provide the practical experiences necessary to adequately apply this 
knowledge, and subsequently qualify to stand these watches. 

2. Test Demographics 

Approximately ten years of historical tests scores, forty classes and over 
5000 students, were analyzed and the outcomes consolidated. Although 
summarized here, appendix C contains the detailed tabulated results. On 
average, each class contained 183 students. Students were required to achieve 
relatively high scores to pass an exam. To pass the Rules of the Road test, 
students must have scored at least a ninety percent. Eighty percent was 
required to successfully pass all other exams and quizzes. Historically, the 
Navigation module proved to be the toughest to successfully pass. This module 
had two tests. On average, 33 students (18 percent) failed the first exam and 38 
students (21 percent) failed the second one. This module also contained a 
graded practical that roughly 84 students (46 percent) in each class failed. 
Students also had historical difficulties passing the Rules of the Road and 
Combat Systems tests. Both of these exams had historical failure rates above 
ten percent. 

As it turns out, unexpectedly, when asked in a survey by IDSI to rate the 
importance of each of the topics at SWOS, Surface Warfare officers 15 highest 
ranked topics are contained in the modules that have the highest failure rates 
among students. A table summarizing the results, which is published in IDSI’s 
Final Report is included in appendix C subsection A. 


66 



3. Effectiveness of the Curriculum 

Research shows that in many instances, students view their teachers, not 
their textbooks, as the primary source of information. This is true, in part, due to 
the fact that students believe that they can rely on their teacher to develop new 
concepts sufficiently well that the associated test can be passed without reading 
the book (Nix and Spiro, 1990, p. 110). It is possible that this strategy served 
students well enough in high school and college, but it may be inadequate to 
facilitate acceptable learning transfer when a student is faced with internalizing 
extensive quantities information that is passively learned over a shorter periods 
of time. 

Remediation for students that failed an exam was carried out through a 
mandatory study program. These students were required to study within the 
schoolhouse for an arbitrarily designated period of time each week, typically 20 
hours. This studying was done on the students’ time after normal working hours. 
This process was used by the school as a necessary motivational tool for 
improving study habits and ultimately, retained knowledge. Since the school 
experienced a very low attrition rate, can the remediation process can be 
classified as a success. This approach appears to have helped students 
prepare for and pass their next exam, but it did not necessarily enable them to 
successfully use that knowledge, or several skill sets taught at SWOS, when they 
first arrived aboard their ships. 

In its final report to the Chief of Naval Education and Training, Intelligent 

Decision Systems, Inc. (IDSI) posted the results of a survey that attempted to 

gain insight into what fleet officers thought were the core competencies required 

to be a successful division officer. The survey asked, “What competencies do 

you believe to be most necessary for a division officer to be effective?” The 

survey also asked, “From your experience, what competencies are most often 

lacking in SWOS graduates?” 22 of 55 (40 percent) of the respondents listed as 

a most often lacked competency at least one of the items that they listed as most 

necessary for division officer success. Comments included such things as 

administrative programs, customs and courtesies, naval platforms and their 
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missions, seamanship, communications, rules of the nautical road, chain of 
command, technical knowledge, and fundamental knowledge of guns, missiles, 
and basic engineering. Comments regarding competency deficiencies in 
seamanship, ship handling, and watch standing were common. 

If only IDSI surveys were used to judge this course of instruction, some 
could arrive at the erroneous conclusion that the program was not successful 
because Ensigns’ initial performance aboard their first ships was lacking in some 
way. The IDSI surveys do not address the number of Ensigns whose 
performance the commanding officer’s had been pleased or displeased with. It is 
possible that when answering the two survey questions dscussed above, the 
respondents had only seen one or two junior officers that arrived aboard with the 
deficiencies listed. It is also possible, that the respondents were answering the 
surveys generally, and found most Ensigns lacking the knowledge or skills that 
were described in the answers. 

Based on my experiences, there may be some merit to the notion that 
some Ensigns lacked in one or more aspects of knowledge or specific skills when 
they graduated from SWOS. However, even if true, the program cannot be 
described as a failure. It is possible, that utilizing a traditional classroom-lecture 
approach, coupled with minimal laboratory training prior to sending new Ensigns 
to their first ships, resulted in an inadequate internalization of knowledge, 
insufficient knowledge transfer, and subsequent deficits in either knowledge or a 
particular skill set that would lead to the comments found in the above mentioned 
survey. It is also possible, and argued by many that I have informally talked to, 
that the program was extremely successful, and only a handful of Ensigns ever 
reported aboard their first ship deficient in either knowledge or a particular set of 
skills. 

Whatever the case, as a rule, once Ensigns checked aboard their first ship 
they received a lot of on the job training (OJT) and spent many hours studying for 
watchstation qualifications. The Ensigns were working with and studying the 
same information already taught to them in SWOS. It is likely the information 


68 



was recalled more easily since it had already been seen once before, processed, 
and stored in memory while at SWOS. However, there may have not been 
enough time to rehearse with the information that was taught in SWOS to 
successfully build semantic mappings. By being afforded the opportunity to 
study the information over a more prolonged period of time, semantic mappings 
were quickly constructed, and in general, the junior officers rapidly became 
knowledgeable and able to satisfactorily perform the tasks that were difficult 
when they first checked aboard. 

4. Personal Experience 

Students had opportunities to practice their budding skills in numerous 
laboratories. Students were able to practice ship-handling, communications, and 
damage control procedures in a relatively benign environment. Since student 
egos were the only things ever broken, these labs proved to be a positive training 
experience. However, time in these trainers was limited, and the best that could 
be expected from a novice student was basic familiarity with the concepts and 
principles being taught. 

5. Conclusions About the Traditional SWOS Curriculum 

The traditional SWOS division officer curriculum possessed defined 
learning objectives, a first-rate core curriculum, a feedback mechanism that 
effectively served its purpose, good assessment execution, and an established 
remediation process. It is possible that since large quantities of information was 
taught very rapidly, that some Ensigns could not process and rehearse with that 
information long enough to shift it into long-term memory and subsequently, into 
a semantic mapping. As a result, some of the Ensigns’ knowledge within the 
realm of nautical instruction remained inert. Many times, out of necessity. 
Ensigns focused on and memorized enough facts to pass their next exam. 
These facts, however, tended to be easily forgotten, and so the knowledge 
remained “inert” (Nix and Spiro, 1990, p. 123). 

If the new training initiative is going to receive good reviews from the fleet, 
an instructional methodology is needed that helps the students use multiple facts 
in constructive problem solving environments so that not only are facts 
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remembered, they become useful tools. Curriculums that utilize delivery 
methods that incorporate learning activities within the context that these activities 
normally occur, along with a presentation of the culture in which they occur, will 
best achieve the goal of knowledge transfer (Brown, Collins, and Duguid, 1989, 
p.33). 

C. FIELD OBSERVATIONS 

I am able to make several personal observations regarding this training 
initiative. These observations are in addition to survey data, but include 
information learned during post-survey discussions. Collecting survey data and 
making self-reporting field observations enables the viewing of information from 
multiple perspectives, and allows the verification of the specific field observations 
and survey data, to what extent each agrees with the other. Four general 
observations are made. First, implementation problems exist in every ship on the 
waterfront. Second, ship Training Officers are, with respect to the Division 
Officer At Sea training initiative, very frustrated. Third, the students are 
discouraged about the Division Officer At Sea training initiative. Forth, the 
training initiative’s two stated goals are not well defined ^. 

1. Implementation 

Every ship I visited was currently experiencing implementation problems. 
One ship had not begun implementing the training program. Two ships had only 
one stand-alone computer that was used for this CBT. Several ships had 
installed the courseware on their local area network (LAN), but spent months 
troubleshooting bugs, and were still experiencing problems at the time of my visit. 
None of the training officers I talked to reported that their ship was not 
experiencing difficulties with implementation. 

The manner in which this initiative has been locally implemented differs in 
two discernable ways than was discussed at the first kick-off meeting in May 
2002. At this meeting, it was generally agreed that the Ensigns were not to be 
given division officer responsibilities until after they had qualified Officer of the 

^ 8 ships were visited. Post survey discussions with training officers aiso inciuded the 
impiementation probiems of the other ships aiong the waterfront. 
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Deck (Underway). This qualification has historically taken between 12 and 18 
months to complete depending upon the officer’s abilities and ship’s schedule. I 
observed that many of the junior officers enrolled in this program, most having 
been attached to their ships for less than three months, were assigned significant 
divisional responsibilities. Second, there was after a lot of debate in this kickoff 
meeting, a general consensus that the Ensigns would need computing resources 
supplied to them. The issue at the meeting was whether to directly supply each 
Ensign with a laptop computer or supply ships with computers that were to only 
be used in conjunction with this program. To date, neither has occurred and the 
Ensigns are falling behind schedule due to a lack of computer resources. 

2. Training Officers 

This process frustrates Training Officers. Training Officers believe that 
they were delivered a poor beta product and asked to implement it without 
sufficient instructions, training, or support from any of the other agencies involved 
with this program. 

3. Students 

Every Ensign I talked to appeared to want to succeed. Several, however, 
felt doubtful about their future success in this program because of the 
competition for resources and OJT time on the bridge. The students are 
competing for resources. Some ships have 8 to 10 Ensigns working their way 
through this training initiative. Ships do not have enough computers to study at. 
There are not enough technical manuals, publications, or instructions available to 
simultaneously support shipboard operations and student learning. Ensigns are 
being given division officer responsibilities, which occupy a tremendous amount 
of their time and significantly reduces the time available for quality learning. 

OJT time is reduced for each Ensign. Ships are underway for limited 
periods of time each quarter. Ships are becoming overmanned with junior 
officers, and as a result. Ensigns are getting a smaller share of an already limited 
resource. 
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4. Training Initiative’s Goais 

The Division Officer at Sea Training Program Ship’s Handbook defines the 
program goals. They are: 

(1) To accelerate and enhance Surface Warfare Division Officer training so that 
the navy will produce better prepared leaders in less time (p. 1 -3). 

(2) To retain naval officers as career professionals (p. 1 -3). 

These two statements are vague and read more like slogans than 
quantifiable goals. For example, how will accelerated or enhanced training 
produce better leaders? Similarly, is it the increased retention of one or one 
hundred junior officers that will fulfill the second program goal? It appears there 
are no established rubrics, or standards, which have been provided to 
commanders to assist them in determining whether the program goals are being 
met at the shipboard, squadron, and fleet levels. 

Optimally, educational objectives are derived from the program’s goals. 
However, in this instance, the learning objectives are taken from the appropriate 
Personnel Qualifications Standards (PQS) books. A PQS book identifies the 
knowledge and skills that each officer must possess in order to stand the 
shipboard watch that the book was written for. Consequently, the program’s 
goals have no bearing on this training initiative. 

D. SURVEY DATA 

Survey data reveals several strengths and weaknesses currently resident 
within the Surface Warfare School Command’s Division Officer at Sea training 
initiative. Based upon survey data, program strengths include Strong 
Educational Objectives and a strong Core Curriculum. The Evaluation 
Framework surveys identified three significant weaknesses in this training 
initiative. The Instructional Evaluation category’s Assessment function, the 
Program Operation function, and the Instructional Presentation function have 
significant weaknesses. 
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1. Program Strengths 

Based upon survey responses and post survey interviews, the course’s 
primary strengths are its educational objectives and core curriculum functions. 

a. Educational Objectives 

Despite having ill-defined program goals, the Educational 
Objectives are one of the strengths of this initiative. This is to be expected since 
designers of this CBT course simply transferred the course content that used to 
be taught in the classroom to CD-ROM. Because the traditionally taught 
curriculum had strong educational objectives and the information was relevant, 
current, and useful, it stands to reason that the CBT course would also possess 
similar characteristics. 93.8 percent of students agree that learning objectives 
are stated at the beginning of each lesson. Over half of the students confirm that 
the learning objectives are covered by instructional material. It can be 
concluded, therefore, that the courseware is teaching the information the Navy 
has deemed important since lessons are covering the learning objectives, which 
are derived from POS. 

Despite the Educational Objectives function strengths, the topics’ 
learning objectives can be stated better. Nearly half of the students identified 
words or phrases in the learning objectives such as understand, list, explain, give 
an example of, describe, identify, recall, master, or differentiate. These words or 
phrases identify objectives in terms of what the teacher wants the student to do. 
They do not describe student performance in terms of what the student must do 
to satisfy an objective. When a student references the learning objectives as a 
checklist to verify that he or she knows all of the important material, it makes it 
easier on the student if these objectives are written from the student’s 
perspective. 

b. Core Curriculum 

The Core Curriculum is this program’s major strength. What is 
taught in this course is relevant, current, and useful. As previously mentioned, 
course content was transferred to a digital format. Information was neither to be 
added nor deleted during this transformation process. In its traditional format, 
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this curriculum was successfully taught to over 5,000 students from 1992 to 
2002. It is the information in this curriculum that forms the foundation upon which 
an officer’s professional knowledge is laid. 

Over 80 percent of surveyed students believe the presentations are 
relevant to the associated topic and will help them perform their jobs better. 
Student comments regarding the instructional information is very positive. 
Multiple students stated that they believed the information is very pertinent, 
beneficial, and informative. Students also state they believe the information will 
positively reinforce what is learned through OJT. 

75 percent of surveyed Training Officers felt the material is current 
and relevant to today’s work environment. Additionally, the majority of the 
Training Officers believe the instructional material will help students perform their 
jobs better. In fact, one Training Officer reported that the drawings and diagrams 
included in this course are so good that they are now being used for ship-wide 
training. 

2. Primary Weaknesses 

Based upon survey responses and post survey interviews, this program’s 
primary weaknesses are the Instructional Evaluation section’s Assessment 
function, the instructional presentation function, and the program operation 
functions. 

a. Assessment Function 

The Assessment block is rated poorly because ships are 
experiencing tremendous difficulties with the courseware’s testing module. 
Testing is a significant component to any training program. It is vital that these 
technical difficulties be investigated and corrected as soon as possible. Since I 
was unable to survey a student or supervisor that had successfully taken or given 
a module test, I am unable to adequately assess the function and rate it 
accordingly. Consequently, no further comments will be made concerning 
detailed analysis in this area. 
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b. Program Operation Function 

The program operation function is rated low for numerous reasons. 
First, during follow-up interviews students and supervisors unanimously stated 
that the software is difficult to install and set up for use. Second, ships are 
experiencing compatibility issues with the software. Third, security policies may 
leave personal information at risk for compromise. Fourth, improvements can be 
made to the software’s graphical user interface (GUI). 

1. Installation Difficulties. The installation and set-up 
portions of this CBT software need to be automated. 100 percent of training 
officers believed that this CBT courseware is difficult to install and use. The 
majority of surveyed students believed the software was difficult to install or 
selected Not Applicable, since they did not install the software or had someone 
else install it for them. Installation is not intuitive, even to more experienced 
computer users. Some users, like myself, received three CD’s containing the 
entire course. Others received up to six CD’s. A few people were able to 
download the courseware from a website. Installation currently involves 
unzipping files into various folders and then searching through these folders to 
find a set of complicated instructions for adding students and using the software. 
Zipping and unzipping are terms that describe the compression and 
decompression of data. By zipping files, more files can be stored in a single 
location. Flowever, zipping a file renders it unusable until it is unzipped. 

All commercial software with multiple CD’s includes an 
automated installation process. When a person inserts the first CD into the 
computer, they will have to answer a short series of questions and then the 
software automatically installs itself. The users have only to then remove and 
insert requested discs as the program is installed. 

2. Software Compatibility. Ships are also reporting minor 
compatibility issues. The SWOS CBT courseware requires the use of Microsoft 
2000 series of products to function properly. Many of the ships I visited did not 
have the required LAN infrastructure to support the courseware’s Microsoft 2000 
series requirement. Due to increasingly tight budget constraints and licensing 
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issues, it seems doubtful that these ships will start utilizing the required Microsoft 
products in the foreseeable future. 

3. Security Policies. Training Officers report that all student 
data is stored on a computer, or computers, that have direct access to the 
Internet. Even if data is stored on a computer that resides behind a firewall, it is 
well known that the data is not secure if that computer has direct access to the 
Internet. There are two possible solutions for quickly and inexpensively 
improving security and lowering the risk of compromising sensitive information. 
First, move the data to a computer that cannot be connected to the Internet. This 
is known as air-gapping. Transmit and receive the data via computers that are 
connected to the Internet, but move it by floppy diskette to an air-gapped 
machine for storage. This is what most, if not all, major banks have resorted to in 
order to decrease the risk of compromise. The second option, and the one I 
personally favor, is to encrypt all student data and leave it encrypted while it is 
stored on the computer’s hard drive. There are very strong encryption algorithms 
available that cannot easily be broken using even advanced techniques that 
would serve this purpose very well. 

4. Graphical User Interface. During post survey interviews, 
the students made two recommendations for improving the courseware’s GUI. 
Although mixed results appear on the survey (40 percent said they could not 
track lesson progress), students desire a graphical representation of their 
progress through each lesson. This can be accomplished with a graphical dot on 
a sliding bar across the top of the computer screen. It will help the students to 
judge how far they have come in each lesson and my serve as a positive 
motivator to continue through the end of each lesson. The second 
recommendation is to link all the related lessons in such a way as to make them 
accessible from the end of all of the lessons that are related. Many times, a 
group of three or four topics will be completed by the students and they have to 
exit out of the GUI to select another related topic in the same module. The 
students stated this was a source of irritation that served to demotivate them over 
time. 
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c. Instructional Presentation 

The instructional presentation function contains eight elements. 
Four of these elements, Text, Questioning, Lesson Sequencing, and Color, 
received favorable marks by the majority of those surveyed. Unfortunately, three 
of the four remaining elements. Interactivity, Environment, and Instructional 
Quality, received poor marks by those taking the survey. 

1. Interactivity. A highly interactive lesson is one that will 
engage the students, raise their curiosity and interest, and positively motivate 
them to learn. It does this by forcing students to act upon new information 
multiple times, significantly increasing the probability that the information 
becomes resident in long-term memory and semantic maps. 76 percent of 
students disagreed with a survey item asking if lessons kept their attention. 
Additionally, the same percentage of students disagreed with a survey item 
asking if the lessons were entertaining. 11 of the 17 students reported that they 
were frequently bored while using the instructional software. The students 
complain that there is too much reading that has to be done on screen. 

Students who are bored and uninterested are more likely 
than motivated students to page through lessons as fast as they can and devise 
other methods for learning the material, or worse, figure out how to survive 
without learning the material. Because of this, it is important that the 
presentations be improved, making them more interactive and entertaining to the 
students. A previous chapter details the benefits of multi-modal presentations. 
The increased use of audio narration, movies, video clips, and more multi¬ 
layered interactive pictures in conjunction with the textual content is a good place 
to start improving the lessons. 

2. Environment. This element receives a low grade for two 
reasons. Students are having a difficult time finding instructional resources that 
are available long enough to complete most lessons. Additionally, students are 
often interrupted or called away in the middle of lessons. 

Every student I surveyed told me that resources were 
scarce. 70 percent of surveyed students said that they did not have immediate 
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access to instructional resources such as computers, software, books, 
instructions, or manuals. As previously mentioned, computers are in high 
demand and short supply. Ship’s business takes priority over Ensign training 
time. According to students, ships have not received any more manuals, despite 
now being burdened with training newly commissioned officers. While underway, 
manuals that are needed to study are also needed by the watch standers in 
performance of their duties. Consequently, students do not get to study many of 
the manuals while a ship is underway. In the traditional classroom curriculum, 
each student was provided a safe containing a copy of every manual that was 
needed to learn from. 

Only two of surveyed students have been able to start and 
complete lessons without interruption. Over half of the students report that they 
are frequently called away in the middle of lessons. It is difficult for students to 
take in new information, move that information through their STSS into working 
memory, and act on it long enough to store it in long-term memory, if they are 
consistently being interrupted or called away in the middle of their studies. 

65 percent of students and 75 percent of supervisors feel 
that a different location can or should be used for learning with this CBT 
software. The students and their training officers contend, and I agree, that there 
should be a centralized location, or learning resource room, on the waterfront 
where they can go to study while their ship is in its homeport. This would allow 
students learn in a quiet, distraction free environment while taking the CBT 
lessons. When finished, the students can go back to their ships and receive 
reinforcing training via practicums or walk-throughs. 

3. Instructional Quality/Effectiveness. Comparing the 
strengths and weaknesses within the Pedagogical Evaluation and Instructional 
Evaluation sections determines the instructional Quality and effectiveness of the 
CBT initiative. The following paragraphs summarize the previously discussed 
key points that are relevant to this function’s rating. 

The strength of this initiative rests squarely in its core 
curriculum function. The information that is being taught to the students is 
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relevant, current, and useful. However, this may not be enough to ensure the 
success of this program. Initial difficulties that Ensigns are experiencing are not 
making a favorable impression of the warfare community on them. These 
Ensigns’ difficulties and initial impressions of this training initiative may ultimately 
lead to decreased retention within the community. 

The lessons lack needed elements of excitement and 
interactivity. The Ensigns report that they are often bored. The students have 
many duties, often sacrificing needed sleep to take a CBT lesson, and are 
frequently disappointed by the way the information is delivered. Additionally, 
students are having a difficult time finding available computers to use, and when 
they are able to use one, are frequently interrupted or called away. 

Students must often rely on department heads, peers, and 
watch standers for their instruction. These people are all very busy, and do not 
necessarily have the time required to properly teach a concept or skill. 
Additionally, these sailors may not have any instructor training or have the 
requisite ability to effectively teach or train others. For example, the junior 
Quartermaster standing Quartermaster of the Watch on the ship’s bridge, may 
not have the time or the ability to thoroughly explain and answer questions about 
set, drift, dead reckoning, and aspects of celestial navigation to an Ensign the 
comes to the bridge for training. 

The students also Qne of the most telling data points is that 
all of the Training Qfficers believe that the program does not reduce the time it 
takes to learn Surface Warfare Qfficer skills. If learning time is not reduced how 
can the first program goal ever be achieved? As a result of the above 
discussion. Instructional Quality and Effectiveness is rated as very low. 
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V. RECOMMENDATIONS 


A. FURTHER RESEARCH 

IDSI delivered this CBT initiative to the Navy early this year. The majority 
of new accessions into the Surface Warfare community are May and June 
graduates. Consequently, most new students are just starting their course of 
instruction with this CBT software. To date, nobody has fulfilled all of the 
requirements of this program and earned their Surface Warfare Pin. It is 
recommended that this program be tracked and compared to the traditional 
classroom course of instruction in two broad areas. 

1. Knowledge Value Added Analysis 

First, conduct a Knowledge Value Added analysis on the two curriculums 
to verify that this CBT initiative is generating an appropriate return on investment. 
If not, this process will allow evaluators to identify which processes in the 
program need to be modified or eliminated in order to then generate appropriate 
returns on the surface warfare community’s investment. Since this program is in 
its infancy, and has no graduates, there is currently not enough available data to 
support an analysis of this kind at present. 

2. Track and Compare Student Progress with Historical Data 

Second, track the students’ progress from the time they arrive at their first 

ships to the time they earn their CIC Watch Officer and Officer of the Deck 
(underway) qualifications and also their warfare pins. Compare these results to 
the historical qualification data with students that graduated from SWOS in 
Newport, Rhode Island. 

B. NON-SOFTWARE RELATED RECOMMENDATIONS 

Following a review of personal observations and an analysis of survey and 
interview data, three recommendations are provided. First, it is recommended 
that the primary surface warfare training activity on each coast be directed to 
conduct thorough evaluations of ships’ implementation and execution progress 
as long as Ensigns are trained via this program. For example, it is recommended 
that the Atlantic Training Group (ATG) be designated as the East Coast’s training 
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command responsible for evaluating ships’ implementation and execution of this 
CBT initiative. Second, It is recommended that a waterfront study space/learning 
resource center (LRC) be established in every fleet concentration area so that a 
distraction free, off-ship learning environment is created for all students in this 
program. Third, it is recommended that increased resources, including 
computers. Navy instructions, publications, and technical manuals be provided to 
all ships for the exclusive use of junior officers enrolled in this program. 

1. Include Training Commands 

Involving the primary training commands on each coast accomplishes 
several things vital to the success of this program. First, it places much needed 
help and assistance near the waterfront on both coasts. The training officers 
complained that they have been unsuccessful getting the help they need to fully 
implement this program. Personnel at the training command would be trained 
how to properly install and run the software, and would render assistance to 
nearby ships when it is needed. A Lieutenant Commander, with the assistance 
of two or three Lieutenants, and a handful of skilled enlisted specialists at each of 
these training commands would be able to perform the requisite tasks associated 
with this CBT initiative. 

Second, by empowering the training commands to conduct annual or 
semi-annual program evaluations, it ensures that all Ensigns receive the 
necessary and quality instruction that is needed to transform them into the 
outstanding professional mariners that the Navy is known for. 

Third, it allows waterfront seminars to be conducted on a routine basis 
with little expense to the government. These seminars would allow supervisors 
and students to act collaboratively to solve problems, increase awareness, 
discuss cross-decking opportunities, and improve the quality of junior officers’ 
training regimens. 

2. Create Waterfront Learning Resource Centers 

The creation of a waterfront learning resource center is very beneficial for 

students in this program. At present, students are finding it very difficult to 

complete lessons without interruption. A quiet, distraction free environment will 
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improve retention, and subsequently, learning transfer, among the students. 
Additionally, instructional resources would be in a centralized, easily 
maintainable location. In times of fiscal constraint, a twenty-computer LAN that is 
located in an LRC is less costly than two or three computers placed on every 
ship, and is much more economical than buying a laptop for every student. 
Finally, a LRC allows students to work together and learn collaboratively, which 
research has shown will lead to improved retention and performance. 

3. Increase Instructional Resources 

It is important that students have nearly ubiquitous access to computers 
while they are enrolled in a mandatory self-paced Computer-based training 
program. There are not enough computers, publications, and other instructional 
material on-hand to ensure that the students receive the same quality of training 
as was received when it was conducted in Newport, Rhode Island. Not supplying 
the students with the resources necessary for successful completion of the 
program is akin to shooting yourself in the foot. Since these officers are the 
future of this warfare community, it is important that the Navy supply them with 
the necessary tools to complete this course of study. 

C. SWOS DOC CBT SOFTWARE RELATED RECOMMENDATIONS 

Three recommendations related to the software are provided. First, and 
most important, it is recommended that the courseware presentations be 
modified and made more interactive and exciting. Second, it is recommended 
that the software be modified to encrypt all data prior to saving it on either a 
portable media or fixed hard drive. 

1. Improve Instructional Presentations 

It is important that the lesson presentations be improved. Students 
unanimously reported that lessons are, despite containing useful information, 
very boring. More interactive activities need to be included in the lessons. 
These include, but are not limited to numerous questioning techniques, point, 
click and drag activities, and video segments that offer students a choice at their 
end. Each choice starts another video segment, so the student can see and hear 
the consequences of the choices made. To make the lessons less dull, more 


83 



video, animation, interactive pictures, and audio narration should be 
incorporated. Many successful CBT training packages include audio narration. 
Additionally, audio narration makes presentations bimodal, and as mentioned in 
a previous chapter, improves student retention. By improving the lessons’ level 
of interaction and making them exciting, student curiosity, interest, and 
motivation will be improved. As a result, more information will be acted upon, 
memory function will increase, and more transfer will occur. The end result will 
be better-trained officers that are able to apply more of what they learned via the 
CBT software to very complex real-life situations. 

2. Add an Encryption Standard 

By adding an automated encryption scheme to the CBT software package, 
the Navy will significantly increase the safety of personal and performance 
information stored on its computers. Students’ personal and performance data 
will be collected, transmitted, and warehoused multiple times throughout the 
duration of this training initiative. By keeping the data encrypted, and decrypting 
it only when reading or performing maintenance on the files, hackers and other 
individuals with malicious intent will be unable to decipher, read, alter, or use any 
data that may be intercepted or otherwise compromised. 

D. CONCLUSION 

By implementing the recommendations that are presented here, the Navy 
will transform this CBT training initiative from a lackluster performer into a shining 
example of what an organization can accomplish through the use of Computer- 
based training that is reinforced through real world experience while on the job. 
Implementation of the following recommendations will achieve the following: 

(1) Student interest and motivation will improve. Learning transfer will improve, 
and consequently, performance will improve. 

(2) The programs objectives will be met because 

(a) Qualification times for all watch stations will be reduced. As a result, the 
program’s first stated goal of better-prepared Surface Warfare Officers in less 
time will become a reality. 
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(b) An increase in qualified officers amounts to more watch sections while 
underway. As a result of the increase in watch sections, each officer has more 
time off to pursue other qualifications and interests. As a consequence, morale 
among the junior officers will increase, leading to better retention. This satisfies 
the program’s second goal of improving retention rates among in its officer corps. 
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APPENDIX A. STUDENT AND SUPERVISOR SURVEYS 


A. STUDENT SURVEY 

STUD ENT’S SURVEY OF INSTRUCT ION 

Please mark you r choices by pi acing a checkmark in the app ropriate box for each of the 
following items. Your specific answers will remain completely anony mous, but your 
views, in combination with those of others, are extremely important. 

SECTION 1: YOUR BACKGROUND AND USE OF COMPUTERS 

1. What is your ag e? 

Unde r 25.• 26 to 30.• 31 to 35.• 36 to 40.• 

41 or older.• 

2. What is your Gender? 

Male..• Female.• 

3. What is your primary position with respect to t his training initiative? 

Student.• Instructor.• Administrator / Supervisor.• 

4. How would you rate your computer skills? 

Novice or Beginn er.• Intermediate.• 

Advanc ed or Expert.• 


SECTION II: INSTRUCT! ONAL QUALITY ASSESSMENT 


Please circle the number that best represents your view regarding the content of the 
SWOS DIVISION OFFICER COURSEWARE . 

If a particular item is Not Applicable, please circle NA. 

If you Strongly agree with the statement, please circle the Five (5). 

If you agree w ith a statement, please circle Four (4). 

If you neither agree or disagree with an item, please circle Three (3). 

If you Disagree with a statement, please circle T\to (2). 

If you Strongly Disagree with a statement, please Circle One (1). 


1. I unde rstood the pu rpose of the cou rse 

2. The learning objectives were clearly stated 

3. The information that was presented was relevant to the topic 


NA 

NA 

NA 


1....2....3....4....5 


.... 1 .. 

..2.. 

..3.. 

.A.... 

.... 1 .. 

..2.. 

..3.. 

.A.... 
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4. Learning objectives were li sted at the beginning of 
every lesson 

5. AU of the learning objectives were covered by the 
instructional material 

6. Learning objectives include key words or phrases such as 
create, install, set-up, start, repair, diagno se, troubleshoo t, 
organize, or write 

7. Learning objectives include key words or phrases such as 
unde rstand, list, exp lain, give an example of, describe, 
identify, recall, master, or differentiate 

8. AU of the information that was required to satisfy stated 
performance criteria was provided in the lessens 

9. The information I learned will help me perform better 

10. I found the material in this course very useful 

11. Test questions were based upon the learning objectives 

12. Test questions only covered kerns I had been tau^ t 

13. Directions for each test were clear 

14.1 unde rstood the performance standard for each test or quiz 

15.1 unde rstood the performance standard for 
each graded activity 

16. Tests were fairly and properlyproctored / administered 

17.1 received tim ely feedback about my test scores 

18. The feedback I received regarding tests was valuable 

19. The feedback I received regarding my test scores 
wUl help me improve my performance 

20. Graded activities were fairly and properly administered 

21.1 received timely feedback regarding my graded activities 

22. The feedback I received regarding my graded activities 
was valuable 

23. The feedback I received regarding graded activities wiU 
help me improve my performance 

24. The software wa s easy to instaU 

25. The software wa s easy to use 

26. The software menu was easy to find 


NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 

NA....L...2....3....4....5 
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27. The software wa s easy to navigate 

28. Direetions were easy to find 

29. Direetions were easy to follow 

30. On-sereen options were easily und erstood 

31. The software had an integrated seareh funetion 

32.1 always knew where to look for on-screen help 
33. Human help was always available 

34.1 was always ab le to page forward 

35.1 was always ab le to page back 

36.1 was able to bookmark my location and comeback later 

37.1 was able to track my progress though each lesson 

38.1 was able to track my progress through the training program 

39.1 was able to easily gene rate practice tests 
40. Each lesson kept rry attention 

41.1 though t the lessons we re entertaining 

42. The mul tim edia used in each lesson was r elevant 

43. Lessons incorporated audio narration 

44. Hi^-resolution graphics / pictures were incorporated 
into each lesson 

45. The use of relevant video, animation, andp ictures were 
incorporated into each lesson 

46. The use of video, animation, and pictures in eachl esson 
kept rry interest 

47. There were too many videos, animations or pictures 
in each lesson 

48. The information ineach lesson wa s worthwhile 

49. The con rseware taugh t me useful information 

50. After using the courseware, I understand the material better 

51.1 was tfequently bored while using the instructional 
software 

52. The use of color enhanc ed the instructional presentations 

53. Contrasts in the use of co lor made text easier to read 


NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 
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54.1 always had access to a quiet, comfortable, 
distraction free learning env ironment 

NA....1....2....3....4....5 

55.1 was frequently interrupted or called away while using 
tie cour seware 

NA....1....2....3....4....5 

56.1 was able to start and complete a lesson without 
intenup tion 

NA....1....2....3....4....5 

57.1 always had immediate access to instmctional resources 
such as computers, software, books, instructions, 
or manua Is 

NA....1....2....3....4....5 

58. A different location can or should be used fo r learning 
via this CBT software 

NA....1....2....3....4....5 

59.1 was able to easily read textual information 

NA....1....2....3....4....5 

60.1 had to scroll down through the text that was on the screen 

NA....1....2....3....4....5 

61. The que stions I was asked focused on information I had 
already been taught 

NA....1....2....3....4....5 

62. The que stions I was asked focused on information that 

I was about to learn 

NA....1....2....3....4....5 

63. The que stions I was asked were related fo the topic’s 
learning objectives 

NA....1....2....3....4....5 


SECTION ni: SHORT ANSWER 


Please legibfy print youranswers to the following questions. There are no correct or 
incorrect re spouses. However, specific and detailed answers are appreciated. If a 
question is not applicable to your situation, either leave the answer blank or use 
N/A. Your specific answers will remain completely anonymous. 


1. Have you ever failed to meet the required standards for an activity or a test? If yes, 
why do you believe that you failed to meet the required standards? 


2. If you an swered yes to the above question, did you receive remediation? If yes, why 
was the remediation you received either effective or ineffective? 
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3. What are the best parts of this course and why ? 


4. What are the worst parts of this course and why? 
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5. What should be done to improve this eourse of instmetion? 


THANK YOU FOR TAKING THE TIME TO COMPLETE THIS SURVEY. 
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B. SUPERVISOR SURVEY 


SUPERVISOR’S SURVE Y OF INSTRUCTION 


Please mark you r choices by placing a checkmark in the appropriate box for each of the 
folbwing items. Your specific answers will remain completely anony mous, but your 
views, in combination with those of others, are extremely important. 


SECTION 1: YOUR BACKGROUND AND USE OE COMPUTERS 

1. What is your age? 

Under 25.• 26 to30.• 31 to 35.• 36 to40.. 

41 to 45 .• 45to50.• Over50.• 

2. What is your Gender? 

Male.• Female.• 

3. What is your primary position with reject to this training initiative? 

Student.• Instructor.• Administrator / Supervisor. 

4. How would you rate your computer skills? 

Novice or Beginn er.• Intermediate.• 

Advanc ed or Expert.• 


SECTION II: INSTRUCTIONAL QUALITY ASSESSMENT 


Please circle the number that best represmts your view regarding the 
SWOS DIVISION OEEICER COURSEWARE. 

If a particular item is Not Applicable, please circle NA. 

If you Strongly agree with the statement, please circle the Live (5). 

If you agree with a statement, please circle Eour (4). 

If you neither agree or disagree with an item, please circle Three (3). 

If you Disagree with a statement, please circle Two (2). 

If you Strongly Disagree with a statement, please Circle One (1). 


1. The SWOSCOLCOM Division Officer At Sea Training 

Initiative’s goals are well defined NA.... 1....2....3....4....5 

2. I know where the SWOSCOLCOM Division Officer At Sea 

Training Initiative’s goals are defined NA.... 1....2....3....4....5 
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3. The Training Irdtiati’se’s goals do not confli ct with other 
organ izaticna 1 go als 

4. This eourse reduees the students’ OJT requirements 

5. This eourse redue es the time it takes studeits to learn 
Surfaee Warfare Offieer job skills 

6. Eaeh topie has assoeiated learning ob jeetives 

7. Learning objeetives are presented at the beginning of 
every lesson 

8. Learning objeetives inelude key words or phrases sueh as 
ereate, install, set-up, start, repair, diagno se, troubleshoo t, 
organize, or write 

9. Learning objeetives inelude key words or phrases sueh as 
understand, list, exp lain, give an example of, deseribe, 
identify, reeaU, master, or differentiate 

10. The learning objeetives support the organizational training 
initiative’s goals 

11. Test questions are derived from the learning objeetives 

12. Student performanee standards are derived from the 
learning ob jeetives 

13. All studentperformanee standards arepublis hed 

14. Students unde rstand the grading eiiteria for eaeh and 
every test and graded activity 

15. Information taugh t to the students is relevant to the topic 

16. The instructional presentations cover all the learning 
objectives 

17. Instmctional material is current and up to date 

18. Instmctional material will help the students perform their jobs 
better 

19. Directions for each test and graded activity are clear 

20. Tests and graded activities are fair 

21. Remediation is available for students 

22. Remediation for failing to meet a standard is etfective 

23. Each test proctor or activity participant other than the 
student is identified in advance 


NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 
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24. Each test proctor or activity participant other than the 
student is trained and qualified for the part of the test 
or activity they are participating in 

25. All tests are given and graded by the computer 

26. For all test items not graded by a computer, the 
criteria for a hilly correct answer is known by the grader 

27. For all test items not graded by a computer, the performance 
criteria for fully correct answers are establis hed 

28. For all test items not graded by a computer, the criteria for 
afiilly correct answer is used to grade a particular test item 

29. For all test items not graded by a computer, the same person 
that writes the test also g'ades it 

30. For all performance based ac tivities, the criteria for correctly 
performed actions is known by the grader 

31. For all performance based ac tivities, the criteria for correctly 
performed actions is always strictly used by the grader 

32. The collection of student test and ac tivity performance 
data enhanc es organ izational performance 

33. Students receive timely feedback regarding their performance 

34. Feedback to students is relevant and usehil to them 

35. Feedback to students either praises them or identifi es a 
weakness and reveals how to correct that deficiency 

36. The software / courseware is easy to install 

37. The software / courseware is easy to us e 

38. Student progress is easily tracked via the courseware 

39. The cou rseware adequately protects personal information 

40. Files whe re perscna lor performance in formation is stored 
are password protected 

41. Personal and performance information is encrypted during 
trans fers between locaticns 

42. Personal and performance data is stored, updated, and 
maintained on a computer that has direct access to the Internet 

43. Personal and performance data is only downloaded on a 
computer that has direct access to the Internet, but stored on 
a computer that has zero Internet access (air gapped) 

44. Students are provided a quiet, distraction free aiv ironment 


NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 
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45. Students are not interrupted or called away while using the 
courseware to learn 

NA....1....2....3....4....5 

46. A different location can or should be used fo r learning 
via this computer-based training software 

NA....1....2....3....4....5 

47. This computer-based training program’s administrators and 
supervisors are all adequately trained 

NA....1....2....3....4....5 

48. Evaluation standards for assessing this computer-based 
training initi ative hav e been establi shed 

NA....1....2....3....4....5 

49. Evaluation standards for assessing this computer-based 
training initiative have been promulg ated 

NA....1....2....3....4....5 

50. The standards for this computer-based training initiative 
are being met or exceeded 

NA....1....2....3....4....5 

51. Eormal standards are established for the presentation of 
this training initiative’s instructional material by means 
other than the computer (watch standers) 

NA....1....2....3....4....5 

52. Instructional presentations of this training initiative’s 
material by means other than a computer is closely 
monitored 

NA....1....2....3....4....5 


SECTION ni: SHORT ANSWER 


Please legibfy print youranswers to the following questions. There are no correct or 
incorrect re spouses. However, specific and detailed answers are appreciated. If a 
question is not applicable to your situation, either leave the answer blank or use 
N/A. Your specific answers will remain completely anonymous. 


1. If the SWOSCOLCOM Division Officer At Sea Training Initiative’s goals can be 
better defined, how would you change them to make them better? 
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2. How are the students’ performance standards for graded activities established? Are 
these standards urdform throughout the fleet? 


3. What is done with the collected student test and activity performance data? 


4. Does the collection of student test and activity performance data enhance or benefit 
organizaticnal performance? If so, how? 


5. Does the organization require periodic evaluations of this training initiative? If so, 
wha t aspects of this Computer-based training initiative are to be evaluated? 


6. What is the procedure for reporting we aknesses or deficiencies that are discovered in 
the SWOSCOLCOM Division Officer At Sea Training Initiative? 


7. If a weakne ss or deficiency is discovered in the SWOSCOLCOM Division Officer At 
Sea Training Initiative, wha t is the maximum tim e that is allotted to correct it? 
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8. Are standards established for the presentation of instructional material via CBT? If 
yes, please explain why o r how those standards have or have no tbeen met. 


9. What is / are the major strengths of this CBT initiative? 


10. What is / are the major weaknesses of this CBT initiative? 
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11. What should be done to improve this course of instruction? 



99 



THIS PAGE INTENTIONALLY LEFT BLANK 


100 



APPENDIX B. COMPLETE SURVEY DATA 


A. RESPONSES TO STUDENT SURVEY, PART II 


NUMBER 

QUESTION 

N/A 

1 

2 

3 

4 

5 

1 

1 understood the purpose of this course 

0 

0 

1 

2 

10 

4 

2 

The learning objectives were clearly stated 

0 

0 

2 

3 

9 

3 

3 

The information that was presented was relevant to 
the topic 

1 

0 

0 

2 

11 

3 

4 

Learning objectives were listed at the beginning of 
every lesson 

1 

0 

0 

0 

10 

5 

5 

All of the learning objectives were covered by the 
instructional material 

1 

0 

1 

5 

6 

3 

6 

Learning objectives include key words or phrases 
such as create, install, set-up, start, repair, 
diagnose, troubleshoot, organize or write 

2 

3 

3 

6 

2 

0 

7 

Learning objectives include key words or phrases 
such as understand, list, explain, give an example 
of, describe, identify, recall, master, or differentiate 

2 

1 

1 

5 

5 

2 

8 

All of the information that was required to satisfy 
stated performance criteria was provided in the 
lessons 

■ 

3 

3 

2 

■ 

0 

9 

The information 1 learned will help me perform my 
job better 

■ 

0 

2 

1 

12 

0 

10 

1 found the material in this course very useful 

1 

0 

1 

5 

8 

1 

11 

Test questions were based upon the learning 
objectives 

5 

1 

0 

3 

6 

1 

12 

Test questions only covered items 1 had been taught 

5 

1 

2 

2 

4 

2 

13 

Directions for each test were clear 

5 

0 

0 

3 

6 

2 

14 

1 understood the performance standard for each test 
or quiz 

4 

0 

1 

2 

6 

3 

15 

1 understood the performance standard for each 
graded activity 

4 

0 

1 

2 

7 

2 

16 

Tests were fairly and properly proctored / 
administered 

10 

0 

0 

■ 

H 

■ 

17 

1 received timely feedback about my test scores 

11 

0 

1 

2 

2 

0 

18 

The feedback 1 received regarding tests was valuable 

11 

0 

0 

2 

3 

0 

19 

The feedback 1 received regarding my test scores 
will help me improve my performance 

11 

0 

0 

■ 

H 

0 

20 

Graded activities were fairly and properly 
administered 

10 

0 

0 

3 

3 

0 

21 

1 received timely feedback regarding my graded 
activities 

11 

0 

0 

2 

3 

0 

22 

The feedback 1 received regarding my graded 
activities was valuable 

11 

0 

0 

2 

3 

0 

23 

The feedback 1 received regarding my graded 
activities will help me improved mv performance 

11 

0 

0 

3 

2 

0 

























































































































































24 

The software was easy to install 

25 

The software was easy to use 

26 

The software menu was easy to find 

27 

The software was easy to navigate 

28 

Directions were easy to find 

29 

Directions were easy to follow 

30 

On-screen options were easiiy understood 

31 

The software had an integrated search function 

32 

1 always knew where to look for on-screen help 

33 

Human help was always available 

34 

1 was always able to page forward 

35 

1 was always able to page back 

36 

1 was able to bookmark my location and come back 
later 

37 

1 was able to track my progress through each lesson 

38 

1 was able to track my progress through the training 
program 

39 

1 was able to easily generate practice tests 

40 

Each lesson kept my attention 

41 

1 thought the lessons were entertaining 

42 

The multimedia used in each lesson was relevant 

43 

Lessons incorporated audio narration 

44 

High-resolution graphics / pictures were 

incorporated into each lesson 

45 

The use of relevant video, animation, and pictures 
were incorporated into each lesson 

46 

The use of video, animation, and pictures in each 
lesson kept my interest 

47 

There were too many videos, animations, or pictures 
in each lesson 

48 

The information in each lesson was worthwhile 

49 

The courseware taught me useful information 

50 

After using the courseware, 1 understand the 
material better 

51 

1 was frequently bored while using the instructional 
software 

52 

The use of color enhance the instructional 
presentations 

53 

Contrasts in the use of color made text easier to 
read 

54 

1 always had access to a quiet, comfortable, 
distraction free learning environment 

55 

1 was frequently interrupted or called away while 
using the courseware 

56 

1 was able to start and complete a lesson without 
interruption 


8 0 


0 


2 10 


7 

3 

0 

2 

0 

2 

0 

2 


0 0 


2 11 2 
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57 

1 always had immediate access to instructional 
resources such as computers, software, books, 
instructions or manuals 

0 

8 

■ 

2 

2 

■ 

58 

A different location can or should be used for 
learninq via this CBT software 

■ 

■ 

2 

2 

6 

5 

59 

1 was able to easily read textual information 

2 

1 

0 

4 

9 

1 

60 

1 had to scroll down through the text that was on the 
screen 

4 

3 

5 

3 

2 

0 

61 

The questions 1 was asked focused on information 1 
had already been tauqht 

■ 

■ 

H 

H 

H 

0 

62 

The questions 1 was asked focused on information 
that 1 was about to learn 

■ 

0 

6 

3 

7 

0 

63 

The questions 1 was asked were related to the 
topic's learning objectives 

■ 

■ 

0 

1 

3 

1 


Table 2. Responses to Student Survey, Part II 


B. RESPONSES TO SUPERVISOR SURVEY, PART II 


NUMBER 

QUESTION 

N/A 

1 

2 

3 

4 

5 

1 

The SWOSCOLCOM Division Officer At Sea Training 

1 nitiative's goals are well defined 

0 

0 

0 

1 

3 

0 

2 

1 know where the SWOSCOLCOM Division Officer At 
Sea Training Initiative's goals are defined 

0 

0 

0 

■ 

3 

0 

3 

The Training Initiative's goals do not conflict with 
other organizational goals 

0 

0 

■ 

■ 

2 

0 

4 

This course reduces the student's OJT requirements 

0 

3 

1 

0 

0 

0 

5 

This course reduces the time it takes students to 
learn Surface Warfare Officer job skills 

0 

2 

2 

0 

0 

0 

6 

Each topic has associated learning objectives 

0 

1 

2 

0 

1 

0 

7 

Learning objectives are presented at the beginning 
of every lesson 

0 

0 

0 

0 

4 

0 

8 

Learning objectives include key words or phrases 
such as create, install, set-up, start, repair, 
diagnose, troubleshoot, organize or write 

0 

0 

0 

0 

■ 

0 

9 

Learning objectives include key words or phrases 
such as understand, list, explain, give an example 
of, describe, identify, recall, master, or differentiate 

0 

0 

0 

1 

3 

0 

10 

The learning objectives support the organizational 
training initiative's goals 

0 

0 

1 

2 

1 

0 

11 

Test questions are derived from the learning 
objectives 

0 

0 

0 

0 

3 

1 

12 

Students performance standards are derived from 
the learninq objectives 

0 

0 

2 

■ 

■ 

0 

13 

All student performance standards are published 

0 

0 

3 

1 

0 

0 

14 

Students understand the grading criteria for each 
and every test and graded activity 

1 

0 

2 

0 

1 

0 

15 

1 nformation taught to the students is relevant to the 
topic 

0 

0 

1 

0 

2 

1 
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16 

The instructional presentations cover all the learning 
objectives 

0 

0 

17 

1 nstructional material is current and up to date 

0 

0 

18 

Instructional material will help the students perform 
their jobs better 

0 

0 

19 

Directions for each test and graded activity are clear 

1 

0 

20 

Tests and graded activities are fair 

1 

0 

21 

Remediation is available for students 

0 

0 

22 

Remediation for faiiing to meet a standard is 
effective 

0 

0 

23 

Each test proctor or activity participant other than 
the student is identified in advance 

0 

0 

24 

Each test proctor or activity participant other than 
the student is trained and qualified for the part of 
the test or activity they are participating in 

0 

0 

25 

All tests are given and graded by the computer 

2 

0 

26 

For all test items not graded by a computer, the 
criteria for a fully correct answer is known by the 
grader 

2 

0 

27 

For all test items not graded by a computer, the 
performance criteria for fully correct answers are 
established 

2 

0 

28 

For all test items not graded by a computer, the 
criteria for a fully correct answer is used to grade a 
particular test item 

2 

0 

29 

For all test items not graded by a computer, the 
same person that writes the test also grades it 

2 

0 

30 

For all performance based activities, the criteria for 
correctly performed actions is known by the grader 

3 

0 

31 

For all performance based activities, the criteria for 
correctly performed actions is always strictly used 
by the grader 

2 

0 

32 

The collection of student test and activity 
performance data enhances organizational 

performance 

2 

0 

33 

Students receive timely feedback regarding their 
performance 

2 

0 

34 

Feedback students receive is relevant and useful to 
them 

2 

0 

35 

Feedback to students either praises them or 
identifies a weakness and reveals how to correct the 
deficiency 

2 

0 

36 

The software / courseware is easy to install 

0 

3 

37 

The software / courseware is easy to use 

0 

2 

38 

Student progress is easily tracked via the 
courseware 

0 

3 

39 

The courseware adequately protects personal 
information 

0 

0 

40 

Files where personal or performance information is 
stored are password protected 

0 

0 


3 

0 

3 

0 


0 0 3 


0 0 2 0 0 


0 0 3 


0 0 


0 0 2 0 0 


0 


2 0 


0 2 


0 0 


0 0 


0 2 0 0 0 


0 0 0 2 0 


0 0 


0 0 


0 0 


0 0 
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41 

Personal and performance information is encrypted 
during transfers between locations 

0 

0 

0 

4 

0 

0 

42 

Personal and performance data is stored, updated, 
and maintained on a computer that has direct 
access to the Internet 

0 

0 

0 

■ 

3 

0 

43 

Personal and performance data is only downloaded 
on a computer that has direct access to the 
Internet, but is stored on a computer that has zero 
Internet access (air gapped) 

0 

0 

1 

0 

0 

0 

44 

Students are provided a quiet, distraction fee 
environment 

0 

0 

2 

0 

■ 

■ 

45 

Students are not interrupted or called away while 
using the courseware to learn 

0 

1 

2 

0 

1 

0 

46 

A different location can or should be used for 
learning via this computer-based training software 

0 

0 

■ 

0 

3 

0 

47 

This computer-based training program's 

administrators and supervisors are all adequately 
trained 

0 

3 

■ 

0 

0 

0 

48 

Evaluation standards for assessing this computer- 
based training initiative have been established 

0 

0 

3 

0 

1 

0 

49 

Evaluation standards for assessing this computer- 
based training initiative have been promulgated 

0 

0 

3 

1 

0 

0 

50 

The standards for this computer-based training 
initiative are being met or exceeded 

■ 

■ 

2 

0 

0 

0 

51 

Formal standards are established for the 
presentation of this training initiative's instructional 
material by means other than the computer (watch 
standees) 

0 

0 

2 

0 

2 

0 

52 

Instructional presentations of this training 
initiative's material by means other than a computer 
is closely monitored 

0 

0 

2 

0 

2 

0 


Table 3. Responses to Supervisor Survey, Part II 

C. RESPONSES TO STUDENT SURVEYS, PART III 

Question 1. Have you ever failed to meet the required standards for an 
activity or test? If yes, why do you believe that you faiied to meet the 
required standards? 

(1) Yes, sometimes test / quiz questions asked for material that was not 
sufficiently covered during the lesson. 

(2) Yes, DC was especially difficult. The test (practical test) asked many 
questions that weren’t in the training. 
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(3) Only on the practical assessments. I believe the cause was too much 
information to go through before being tested. It was hard to retain the info(4) 
Yes, the required standards were never stated. 

(5) Yes for an exercise, no for test. 

(6) Yes, the exams are too detailed with useless information. 

(7) I have not attempted a test yet. 

(8) I have passed all tests taken. 

Question 2. If you answered yes to the above question, did you receive 
remediation? If yes, why was the remediation you received either effective 
or ineffective? 

(1) No, there is no sort of remediation system on this ship. 

(2) No...no support resources were available. Our DC workbooks, for example, 
leaves a lot to be desired. 

(3) Never received remediation because the administration did not know the 
standards, therefore it was ineffective. 

(4) Received immediate remediation for the exercise I failed to meet 
requirements for. Very effective. 

Question 3. What should be done to improve this course of instruction? 

(1) Make a conscious consistent effort to upgrade the software. 

(2) More computers 

(3) Give us a training manual. Give us our own copy of the software if you don’t 
give us a training manual or give us a computer to install it on. 

(4) There should be a room or place where JO’s can work on their training. I 
have no place to do my computer training. 

(5) Reduce the number of practicums and possibly make them more 
consolidated. There is plenty of ways to get the experience and knowledge 
without having to jump through hoops in order to get stuff done. Allowing more 
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time to work on the training would be a good help. I don’t think a division should 
be given a new JO right away. Give him/her time to establish a good routine and 
to get accustomed to the program before having to run a division as well. 

(6) Create an environment where there are computers and time to do it. Make 
installation easier. Create reasonable tests that test common knowledge that we 
will remember. 

(7) Handbooks and slightly clearer instructions on how to maneuver the modules 
and course information. 

(8) Issue every Ensign a computer and a set of discs. Allow more OJT in 
conjunction with the modules. Shorten the modules or increase the amount of 
lessons to decrease the amount of time spent learning. Train the trainers. Give 
them a course explaining the program and how to best implement it. 

(9) Utilize decommissioning FFG’s and CD’s as “school ships” to maximize the 
hands-on training and dedicated training personnel. 

(10) There is no doubt that much thought was put into this program...the only 
criticism I have lies in the program’s organization and execution. There is no 
doubt that ships are not being given the technology equipment and guidance on 
how to carry out the programs intent. Training Officers are not given training on 
how to carry out the program. With time, it will surely improve. 

(11) Send us back to SWOS. I believe it is a needed part of officer training-just 
shorten it some. Until the Navy can dedicate a person to teaching SWOS-at-Sea 
on each ship and train that person first on how to run the program, the program 
will not work. Then ships need the funding to run this; i.e. computers for the 
program and a space to work in. To make this program really effective, the Navy 
needs to put this program at the top of the list for all new Ensigns and their CO’s. 
As long as the Ensigns have command duties, the program will never be 
effective. My personal feeling is that new officers with no prior military 
experience need something prior to the ship that is military in manner. 
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(12) We need more structured instruction and dedicated personnel. Ship’s 
officers are too busy with other primary duties. 

(13) Ensure proper training of system administrators/trainers. Provide computers 
devoted entirely to the SWOS-at-Sea program. Overhaul some of the lessons so 
they provide more relevant info in a more engaging format. Overhaul the tests so 
that they evaluate the material that was presented in the lessons. Solve 
technical problems with the program. 

(14) The program should be reconstructed to be more user friendly. Without the 
help of someone helping me with the initial startup process I am unable to start 
the program at all. 

Question 4. What are the best parts of this course and why? 

(1) I have little experience with the course, but I like the simplicity. 

(2) A lot of the material is pertinent, and beneficial. It is good to be able to study 
this material on your own. Some to the multimedia is interesting and helpful. 
Combining formalized instruction via the CD and the OJT is probably more 
effective than 6 months in a classroom. 

(3) The information is very good, and it does reinforce our practical 
understanding. However, standing 5 and dimes (A type of watch section rotation 
where a watch stander typically stands five hours of watch and then has ten 
hours off before coming back on watch again) underway, it is difficult to summon 
one’s energy to find computer time and complete lessons. We are not supposed 
to be division officers, and I am taking the responsibility for a lot of divisional 
duties because the opportunity is there. 

(4) Some of the lessons have good visuals that help the learning process and 
could not be accomplished with a book or chalkboard. Could be a good way to 
learn if nothing else was going on. 

(5) Interactive videos, sounds, questions. They keep my attention better than 
just textual slides. 

(6) Saves money. Immediate utilization of bodies. 
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(7) The slides are very informative and contain useful information. The 
knowledge is presented in a plain and easily understood manner. The animation 
in the later slides add an element of excitement to the lessons as well as make it 
easier to pay attention and envision real life application of knowledge. 

(8) You don’t have to go to SWOS. Good way of getting information without 
asking people. 

(9) Learn at own pace and ability to work when I have time. It allows flexibility in 
the schedule while still teaching me important information. The info did improve 
my knowledge of the ship and its abilities while seeing them first hand. I know it 
makes a good learning experience. 

(10) The information is helpful. 

Question 5. What are the worst parts of this course and why? 

(1) I have difficulty saving and progress. I don’t know if I am doing it wrong or 
there is a real problem. 

(2) Technical problems with the program. A lack of guidance / instruction. NO 
COMPUTER AVAILABILITY; most of ship’s computers being used for ships 
work. Some of the slides are too long and boring...not presented in an engaging 
format. Some of questions on the quizzes / tests asked for information that was 
not adequately presented in the lessons. The SWOS-at-sea program is not a 
priority for most ships...it is difficult to find officers who have enough time to 
review the material thoroughly. 

(3) We need dedicated computer time and space. Dedicated instructors are 
needed...i.e. shore based learning center (ensuring fairness). Too many officers 
so we don’t actually get enough driving / learning time. 

(4) No human interaction for ready reference questions. The programs often 
don’t work. Ships cannot dedicate time to working on this...there are always 
other things going on that are mission critical for the ship and they take priority. It 
is very difficult to dedicate time to learn when your have 10-12 hours of watch, 4 
hours of meetings, and various other things going on with little or no sleep (A 
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normal underway). Underway is a bad environment for sit down learning. Some 
functions are hard to operate or don’t work at all. To date (over 3 months), our 
testing CD’s are not working. Other ships have reported that the tests are not 
saving upon completion. 

(5) Length of lessons. Mundane textual slides. Difficult to maintain attention 
span, especially while underway and standing watches with little sleep. 

(6) IT problems are rampant. The modules will not open or information will not 
save. Standards were never explained to the trainees, therefore performance is 
low. The ship was never trained in how the program should run. There are no 
dedicated trainers; therefore students are not a priority. The lack of training is 
not complimentary to the profession. Over-manning decreases hands-on training 
at tasks such as ship driving. 

(7) Sitting at a computer reading for hours is very boring. The modules are very 
long and require a lot of time to complete. Procuring a computer for a sufficient 
amount of time to complete a module and learn the material was very difficult. 
Learning is not a priority onboard and outside resources were not readily 
available. OJT is very limited. 

(8) Loss of human interaction leads to a loss of initiative. Hard to stay focused 
and can’t ask questions on specific problems. It is very hard to balance the Divo 
at sea courseware requirements and qualifications. Qualifications tend to take a 
majority of the time which leaves less time with the courseware. 

(9) There are too many slides and too much information to do while being a Divo 
and also learning the same stuff in your PQS. If you don’t have a laptop you 
can’t complete the computer training. Installation needs to be put into a single 
.exe file which is extractable with one click. 

(10) There are too many activities (practicums). While I understand the 
relevance and importance, I frequently find myself lacking enough time to work 
on them. Balancing a division, watches, other PQS, and other aspects of ship 
life doesn’t leave much time to run around looking up references and observing 
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evolutions, as well as cross decking to other ships to see how they do it. Just too 
much to do in too little time. 

(11) The length of each module and how interesting the lessons are. 

(12) I would like hard copy, like a textbook or (reference) TRAMAN for learning 
the material 

(13) Give each person a set of discs. 

D. RESPONSES TO SUPERVISOR SURVEY, PART III 

Question 1. If the SWOSCOLCOM Division Officer at Sea Training 
Initiative’s goais can be better defined, how wouid you change them to 
make better? 

(1) When the system actually comes up on the computer, the goals are easy to 
identify. 

(2) Better printed reference material should be provided. 

Question 2. How are the students’ performance standards for graded 
activities estabiished? Are these standards uniform throughout the fieet? 

(1) The results / answers are given through the program. 

Question 3. What is done with the coiiected student test and activity 
performance data? 

(1) Sent off to SWOS. Nobody has taken a test yet. 

Question 4. Does the coiiection of student test and activity performance 
data enhance or benefit organizationai performance? If so, how? 

(1) Can’t tell yet because nobody has taken a test. The Ensigns are excelling in 
SWO knowledge by OJT. 

Question 5. Does the organization require periodic evaiuations of this 
training initiative? If so, what aspects of this Computer-based training 
initiative are to be evaiuated? 

(1) Everything needs to be evaluated. 
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Question 6. What is the procedure for reporting weaknesses or 
deficiencies that are discovered in the SWOSCOLCOM Division Officer at 
Sea Training Initiative? 

(1) Via Email. 

(2) Email SWOS POC. 

Question 7. If a weakness or deficiency is discovered in eh SWOSCOLCOM 
Division Officer at Sea Training Initiative, what is the maximum time that is 
aiiowed to correct it? 

(1) Until it gets resolved. 

(2) The problem is identified. If computer disc/software discrepancies or missing 
data, and turn around time until receipt of correction CD was around 2 to 3 
months. 

Question 8. Are standards estabiished for the presentation of instructionai 
materiai via CBT? If yes, please explain why or how those standards have 
or have not been met. 

(1) No, zero guidance was given about the program and since it is difficult to set 
up on the LAN, we don’t have a presentation of instructional material. It’s all over 
the LAN and difficult to navigate through. 

(2) No specific guidelines have been established as of yet. Students are 
expected to review modules on their own time with the primary responsibilities 
still being watch stander qualification and divisional obligations. 

Question 9. What is / are the major strengths of this CBT initiative? 

(1) Appears as though decent information is on the discs once we can pull them 
up. 

(2) The concept of qualifying SWOS more quickly. 

(3) Information/diagrams are very useful in illustrating certain theories or systems 
(e.g. damage control, engineering, combat systems) and have been used to 

garner info for larger command wide officer training. 
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Question 10. What is / are the major weaknesses of this CBT initiative? 

(1) Difficult to navigate through. No instructions were given to the training 
officers. It was just sent out to the fleets. Looks like the product had not been 
QA’d because many modules were missing or incomplete. No real 
troubleshooting help is available. Not much guidance. 

(2) The printed reference material is poor. The installation is poorly explained. 
The support to the ships is weak. 

(3) Command shortage of available computer resources to dedicate to solely 
CBT program. Personal aspect eliminated-most instruction comes from CD’s 
and questions or OJT issues cleared up while on watch. Loading CD’s and 
tracking current versions while deployed was difficult. Command emphasis more 
towards PQS program rather than CBT now that SWOS has been essentially 
taken out of the picture. 

Question 11. What should be done to improve this course of instruction? 

(1) Instruct the supervisors before sending out the material. Have teams come to 
the ship and implement it on the LAN and then give lectures on how to navigate 
through. Explain saving and exporting data procedures. 

(2) The CBT lacks Polish. Installation is a pain and there is not any real tech 
support. The CBT is not geared to the ship’s LANs. Have had a hard time 
finding computers that can run the program. We have had a hard time finding 
computers with the appropriate software and hardware. 

(3) Training seminar on the waterfront for implementation and expectations of 
program. With this program coming into the fleet during the beginning of the Gulf 
War, information and program instructions were difficult to ascertain. Hopefully, 
a waterfront training initiative would get everyone on the same page. 
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APPENDIX C. HISTORICAL SWOS DATA 


A. SWOS DOC TOPICS AND THEIR RELATIVE IMPORTANCE 


Topic 

Number 

Topic Titie 

Survey 

Score 

Survey 

Ranking 

1.01 

Review Preliminary Rules of the Road 

3.840 

1 

2.04 

Standard Commands 

3.811 

2 

2.02 

R/T Procedures 

3.809 

3 

5B.4 

Opening CPA/ Avoiding Course 

3.809 

4 

1.03 

Steering and Sailing Rules 

3.784 

4 

5B.1 

Introduction to Maneuvering Boards 

3.779 

5 

5A.5 

UNREP 

3.747 

6 

5B.2 

Tracking 

3.747 

7 

2.19 

Emergency Actions 

3.730 

8 

1.02 

Lights and Dayshapes 

3.722 

9 

2.17 

Man Overboard Procedures 

3.716 

10 

5A.13 

Aids to Navigation 

3.716 

11 

1.04 

Restricted Visibility Sound and conduct 

3.711 

12 

11.18 

Firefighting School 

3.710 

13 

5A.10 

Charts and Publications 

3.684 

14 

5A.20 

GPS 

3.684 

15 

2.11 

Shiphandling Alongside a Pier 

3.670 

16 

2.12 

CONREP Conning 

3.660 

17 

5A.19 

Navigation Detail and Anchoring 

3.650 

18 

5A.2 

Mooring 

3.631 

19 

10.01 

3-M and the Division Officer 

3.628 

20 

11.17 

Buttercup Get Wet Trainer 

3.622 

21 

5B.3 

Changing of Station 

3.621 

22 

5A.1 

Deck Seamanship 

3.600 

23 

10.05 

Equipment Tag-Out 

3.595 

24 

11.12 

Personnel Protective Equipment 

3.590 

25 

2.05 

Communications and the Watchstander 

3.560 

26 

11.11 

Portable DC Equipment 

3.550 

27 

11.07 

Fires and Extinguishing Agents 

3.540 

28 

11.09 

Fixed DC Systems 

3.532 

29 

5B.8 

True/Desired Wind 

3.526 

30 

6.13 

Electrical Safety 

3.526 

31 

3.04 

Honors and Ceremonies 

3.490 

32 

4.12 

Navy Safety Programs 

3.474 

33 

10.07 

Ouarterly and Weekly Schedules 

3.473 

34 

5A.6 

Flight Deck 

3.463 

35 

6.06 

POS System With Practical 

3.457 

36 

5B.5 

Tactics, Columns, and Screen Formations 

3.450 

37 





























































































































































7.09 

Tactical Data Links 

3.426 

38 

11.02 

Compartmentalization and WTI 

3.426 

39 

3.03 

Ship's Emergencies/Security Threats 

3.420 

40 

6.02 

1 DTC and Division Officer Inspections 

3.415 

41 

6.1 

Casualty Reports 

3.409 

42 

11.16 

Practical DC 

3.396 

43 

5A.12 

Tides and Cu rrents 

3.390 

44 

10.06 

Cycle Schedules 

3.383 

45 

11.15 

DC Communications and Symbology 

3.380 

46 

6.08 

Message Formatting ADP 

3.376 

47 

11.19 

CBR-D: Protective Equipment 

3.374 

48 

11.2 

CBR-D: Chemical, Biological, Radiological 

3.374 

49 

11.05 

Portable Dewatering Equipment 

3.372 

50 

3.05 

Small Boats 

3.367 

51 

11.06 

Shoring, Plugging, and Patching 

3.362 

52 

8.05 

U.S. Aircraft and Weapons 

3.362 

53 

8.02 

U.S. Combatants 

3.362 

54 

2.09 

Bridge Equipment 

3.358 

55 

5A.3 

Anchoring and Mooring to a Buoy 

3.358 

56 

2.01 

Surface Combat Information Center 

3.351 

57 

4.05 

JO Fitness Reports and Enlisted Evaluations 

3.347 

58 

5B.6 

Line Formations 

3.337 

59 

2.07 

Underway Bridge Watch Relief 

3.333 

60 

8.03 

Amphibious Ships and Craft 

3.330 

61 

11.01 

1 ntro and Basic First Aid 

3.323 

62 

5A.11 

Basics of Time 

3.305 

63 

10.03 

MDS and the Division Officer 

3.298 

64 

5A.14 

Compasses 

3.295 

65 

4.02 

Enlisted Rating Structure and Advancement 

3.287 

66 

8.16 

Surface Warfare 

3.287 

67 

2.08 

ATP-1 & Publication 102 

3.284 

68 

2.03 

Underway Watch Organization 

3.284 

69 

11.14 

DC Organization and Administration 

3.280 

70 

7.08 

NTDS/CDS/DTE 

3.280 

71 

11.1 

DCA School Lab Tour 

3.272 

72 

6.04 

Shipboard Training Program 

3.269 

73 

8.06 

U.S. Submarines 

3.269 

74 

4.01 

Division Officer Indoctrination 

3.256 

75 

7.02 

Shipboard Communications Systems 

3.255 

76 

8.19 

VBSS 

3.247 

77 

3.02 

Routine/special Evolutions 

3.240 

78 

3.01 

1 nport Watch Organization 

3.231 

79 

4.04 

Sexual Flarassment/Fraternization/Flazing 

3.221 

80 

8.04 

Auxiliaries 

3.215 

81 

5A.7 

Survival at Sea 

3.213 

82 
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8.17 

Air Warfare 

3.213 

83 

6.12 

Operational Messages 

3.210 

84 

7.03 

Shipboard Satellite Communication Systems & 
Equipment 

3.200 

85 

5A.21 

The Future of Electronic Navigation 

3.200 

86 

7.14 

Launching Systems 

3.190 

87 

6.01 

Information Security 

3.180 

88 

7.16 

Cruise Missiles 

3.170 

89 

7.17 

Air Warfare (AW) Missiles 

3.170 

90 

2.14 

ATP & Publication 102 Assignment 

3.160 

91 

8.18 

Undersea Warfare 

3.160 

92 

8.08 

Rules of Engagement 

3.150 

93 

7.15 

Guided Weapons 

3.130 

94 

4.03 

Enlisted/Officer Service Records 

3.090 

95 

6.03 

Schools and Exercises With Practical 

3.090 

96 

7.1 

MK 15 CIWS 

3.090 

97 

8.07 

Intro to Command and Control Systems 

3.080 

98 

2.18 

Navy Publications 

3.070 

99 

7.19 

Sonar Fundamentals 

3.040 

100 

7.06 

1 FF/TACAN 

3.020 

101 

7.18 

AEGIS Weapon System 

3.020 

102 

7.23 

USW Weapons 

3.020 

103 

7.04 

Radar Fundamentals and Systems 

3.010 

104 

11.13 

Basic Gas Free Engineering 

3.010 

105 

5A.27 

Electronic Navigation Practical 

3.010 

106 

2.06 

R/T Crypto System 

3.000 

107 

2.16 

Special CIC Evolutions 

2.990 

108 

7.05 

1 ntroductions to Electronic Warfare 

2.990 

109 

7.13 

Naval Gun Weapon Systems 

2.990 

110 

8.12 

Intro to Amphibious Operations 

2.990 

111 

5A.18 

Sunrise and Sunset 

2.990 

112 

4.06 

Drug and alcohol Abuse 

2.980 

113 

11.04 

Stability 

2.970 

114 

4.07 

Good Order and Discipline/Shore Patrol 

2.950 

115 

4.08 

Manpower Documents 

2.950 

116 

5B.7 

Search Turns 

2.950 

117 

6.05 

C/S 1 nport Training Program 

2.930 

118 

5A.4 

Towing 

2.930 

119 

8.14 

Mine Warfare 

2.880 

120 

8.15 

Amphibious Assault 

2.870 

121 

10.04 

Alterations and Availabiiities 

2.870 

122 

6.15 

CSOSS 

2.860 

123 

7.07 

The Gunfire Control Problem 

2.860 

124 

7.2 

Acoustic Propagation 

2.850 

125 

4.13 

Non-ludicial Punishment 

2.820 

126 


117 


















































































































































































7.12 

Naval Ammunition 

2.800 

127 

7.21 

AN/SQQ-89(V) Part 1 

2.800 

128 

2.15 

Visual Communications 

2.770 

129 

7.22 

AN/SQQ-89(V) Part II 

2.760 

130 

8.01 

Naval Doctrine 

2.750 

131 

2.13 

Subsurface Combat information Center 

2.730 

132 

2.1 

Air Combat Information Center 

2.710 

133 

4.09 

Transfers and Separations 

2.690 

134 

8.09 

Command and Control Warfare 

2.680 

135 

7.01 

Radio Wave Propagation 

2.650 

136 

6.14 

Search and Seizure 

2.590 

137 

7.24 

Ship's Silencing Program 

2.570 

138 

8.11 

Marine Corps 

2.450 

139 

4.1 

Absentees and Deserters 

2.390 

140 

1.98 

Unit 1 Exam 

No Useful 
Data 

141 

1.99 

Unit 1 Exam Review 

No Useful 
Data 

142 

2.2 

Communications Practical 

No Useful 
Data 

143 

2.98 

Module 2/3 Exam 

No Useful 
Data 

144 

3.98 

Unit 3 Exam 

No Useful 
Data 

145 

4.11 

Family Care Policy and Pregnancy 

No Useful 
Data 

146 

4.98 

Unit 4 Exam (Take Home) 

No Useful 
Data 

147 

4.99 

Unit 4 Exam Review 

No Useful 
Data 

148 

6.07 

GENADMIN Messages 

No Useful 
Data 

149 

6.09 

Message Writing Practical 

No Useful 
Data 

150 

6.11 

CASREP Practical 

No Useful 
Data 

151 

6.98 

Unit 6 Exam (Take Home) 

No Useful 
Data 

152 

6.99 

Unit 6 Exam Review 

No Useful 
Data 

153 

7.97 

Unit 7 Exam Preview 

No Useful 
Data 

154 

7.98 

Unit 7 Exam 

No Useful 
Data 

155 

7.99 

Unit 7 Exam Review 

No Useful 
Data 

156 

8.1 

Electronic Warfare 

No Useful 
Data 

157 

8.13 

Naval Surface Fire Support 

No Useful 
Data 

158 
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8.2 

Intelliqence Brief 

No Useful 
Data 

159 

8.98 

Unit 8 Exam 

No Useful 
Data 

160 

8.99 

Unit 8 Exam Review 

No Useful 
Data 

161 

10.02 

Supply and the Division Officer 

No Useful 
Data 

162 

10.08 

PMS Chanqe Manual and Soot Check 

No Useful 
Data 

163 

10.12 

Automated MDS 1 

No Useful 
Data 

164 

10.16 

MDS/PMS Practical Review 

No Useful 
Data 

165 

10.98 

Unit 10 Exam 

No Useful 
Data 

166 

10.99 

Unit 10 Practical Review 

No Useful 
Data 

167 

11.03 

Firemain and Drainaqe Systems 

No Useful 
Data 

168 

11.08 

The AFFF System and Maqazine Sprinklers 

No Useful 
Data 

169 

11.97 

Unit 11 Review for Test 

No Useful 
Data 

170 

11.98 

Unit 11 Exam 

No Useful 
Data 

171 

11.99 

Unit 11 Exam Review 

No Useful 
Data 

172 

12.06 

Simulator Indoctrination 

No Useful 
Data 

173 

12.07 

DRT and Scopehead Plottinq 

No Useful 
Data 

174 

5A.15 

Piloting 1 

No Useful 
Data 

175 

5A.16 

Pilotinq 11 

No Useful 
Data 

176 

5A.17 

Piloting 111 

No Useful 
Data 

177 

5A.25 

Navigation Practical 1 

No Useful 
Data 

178 

5A.26 

Navigation Practical II 

No Useful 
Data 

179 

5A.98 

Unit 5A Exam 

No Useful 
Data 

180 

5A.99 

Unit 5A Exam Review 

No Useful 
Data 

181 

5B.10 

Maneuvering Board Practical Review 

No Useful 
Data 

182 

5B.11 

Maneuvering Board Pre-test 

No Useful 
Data 

183 

5B.12 

Unit 5B Exam 

No Useful 
Data 

184 
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5B.13 

Unit 5B Exam Review 

No Useful 
Data 

185 

5B.9 

Maneuvering Board Practical 

No Useful 
Data 

186 


Table 4. Importance Ratings of Topics from IDSI Final Report (pp. H1-H24) 

B. AVERAGE NUMBER OF FAILURES PER CLASS (NON ENGINEERING) 

The information below was taken directly from the Microsoft Excel files 
maintained at SWOS. This information does not include data from the 
engineering curriculums (Core Phase II). 


Sample Number 

CLASS 

NUMBER 

Unit 1 

Unit 2/3 

Unit 4 

Unit 5 

Unit 6 

1 

109 

7 

12 

0 

25 

0 

2 

110 

17 

4 

0 

34 

5 

3 

111 

24 

33 

0 

41 

7 

4 

112 

0 

2 

0 

9 

0 

5 

113 

0 

29 

2 

18 

9 

6 

114 

18 

29 

1 

32 

7 

7 

115 

17 

28 

2 

31 

34 

8 

116 

28 

10 

1 

45 

5 

9 

117 

5 

3 

2 

27 

5 

10 

118 

9 

14 

3 

34 

3 

11 

119 

24 

2 

3 

26 

13 

12 

120 

15 

13 

5 

72 

4 

13 

121 

9 

20 

2 

77 

10 

14 

122 

Unavailable 

Unavailable 

Unavailable 

Unavailable 

Unavailable 

15 

123 

12 

31 

12 

59 

6 

16 

124 

22 

11 

21 

74 

5 

17 

125 

32 

9 

13 

47 

1 

18 

126 

11 

7 

8 

38 

2 

19 

127 

22 

10 

9 

33 

1 

20 

128 

14 

15 

9 

18 

2 

21 

129 

24 

7 

2 

32 

13 

22 

130 

16 

4 

2 

21 

4 

23 

131 

63 

6 

9 

24 

4 

24 

132 

19 

6 

1 

24 

2 
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Totals: 


Ave Number of 
Failures per class: 


Ave Class Failures 
as a Percentage: 


Sample Number 


1 


2 


3 



Ave Number of 
Failures per class: 


Ave Class Failures 
as a Percentage: 


CLASS 

NUMBER 


109 


110 


111 


112 


113 


114 


115 


116 


117 


118 


119 


120 


121 


122 


123 


124 


125 


126 


127 


128 


129 


130 


131 


132 


133 


134 


135 


136 


137 


138 


Totals: 


642 

628 

650 

261 

319 

22.138 

21.655 

22.414 

9.000 

11.000 

12.12% 

11.86% 

12.28% 

4.93% 

6.02% 


5a exam 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


14 


70 


61 


5b exam 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


35 


60 


62 


Quiz 

Scores 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


68 


76 


74 



46 

61 

50 

70 

65 

109 

34 

44 

54 

37 

30 

50 

34 

28 

44 

10 

16 

49 

17 

16 

83 

9 

12 

54 

14 

17 

77 

12 

19 

65 

25 

29 

119 

30 

38 

115 

16 

56 

160 

48 

69 

122 

78 

44 

110 

8 

14 

117 

633 

715 

1596 

33.316 

37.632 

84.000 

18.25% 

20.61% 

46.01% 
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